Version: 23.4

Pipeline resource optimization

Pipeline resource optimization takes the resource usage information from previous workflow runs to optimize subsequent runs.

The pipeline resource optimization service requires a separate database schema to store its internal data, but also requires access to the Seqera schema. The Seqera and optimization service schemas can coexist on the same database instance.

Docker Compose deployment

Docker Compose makes use of a separate container to set up the pipeline resource optimization service during initialization. Configuration steps differ for new and existing deployments.

New installation

To use the pipeline resource optimization service in a new Docker Compose installation of Seqera Enterprise, use the following steps:

To run the service from a custom URL, declare the URL with the GROUNDSWELL_SERVER_URL environment variable in tower.env. A non-zero value for this environment variable activates the optimization service automatically, so TOWER_ENABLE_GROUNDSWELL does not need to be set when you declare a custom URL.
Set the TOWER_ENABLE_GROUNDSWELL environment variable in tower.env to true. This enables the service at the default service URL http://groundswell:8090.
In your docker-compose.yml file, uncomment the groundswell section at the bottom.
- To create a schema for the optimization service on the same local MySQL container, uncomment the init.sql script in the volumes section.
Download the init.sql file. Store this file in the mount path of your docker-compose.yml file, else update the source: ./init.sql line in your docker-compose.yml with the file path.
When the pipeline resource optimization service is active, pipelines that can be optimized display a lightbulb icon in your Launchpad. Any pipeline with at least one successful run can be optimized.

Existing installation

To use the pipeline resource optimization service in an existing Docker Compose installation of Seqera Enterprise, use the following steps:

To run the service from a custom URL, declare the URL with the GROUNDSWELL_SERVER_URL environment variable. A non-zero value for this environment variable activates the optimization service automatically, so TOWER_ENABLE_GROUNDSWELL does not need to be set when you declare a custom URL.
Set the TOWER_ENABLE_GROUNDSWELL environment variable to true. This enables the service at the default service URL http://groundswell:8090.
In your docker-compose.yml file, uncomment the groundswell section at the bottom. If you use a docker-compose.yml file older than version 23.3, download a newer version of the file to extract the groundswell section.

CREATE DATABASE IF NOT EXISTS `swell`;
CREATE USER 'swell'@'%' IDENTIFIED BY 'swell';
GRANT ALL PRIVILEGES ON *.* TO 'swell'@'%';
FLUSH PRIVILEGES;

If you use Amazon RDS or other managed database services, run the following commands in your database instance:

CREATE DATABASE IF NOT EXISTS `swell`;
CREATE USER 'swell'@'%' IDENTIFIED BY 'swell';
GRANT ALL PRIVILEGES ON `%`.* TO 'swell'@'%';
FLUSH PRIVILEGES;

Download the groundswell.env file. Store this file in the mount path of your docker-compose.yml file. Update the TOWER_DB_URL and SWELL_DB_URL values:

# Uncomment for container DB instances
# TOWER_DB_URL=mysql://db:3306/tower
# SWELL_DB_URL=mysql://db:3306/swell

# Uncomment for managed DB instances (Example URL shows an Amazon RDS instance URL)
# TOWER_DB_URL=mysql://db1.abcdefghijkl.us-east-1.rds.amazonaws.com:3306/tower
# SWELL_DB_URL=mysql://db1.abcdefghijkl.us-east-1.rds.amazonaws.com:3306/swell

When the pipeline resource optimization service is active, pipelines that can be optimized display a lightbulb icon in your Launchpad. Any pipeline with at least one successful run can be optimized.

Kubernetes deployment

Kubernetes deployments use an initContainer that runs during pod initialization to set up the pipeline resource optimization service. To use the service in new or existing Kubernetes installations of Seqera Enterprise, do the following:

Download the groundswell manifest:

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tower-groundswell-cfg
data:
  # Server settings
  SWELL_SERVER_HOST: "0.0.0.0"
  SWELL_SERVER_PORT: "8090"

  # API settings
  SWELL_API_TRAIN_TIMEOUT: "60"
  SWELL_API_TRAIN_BATCH_SIZE: "1000"
  SWELL_API_PREDICT_FRACTIONAL_CPUS: "false"

  # Database settings, different from the tower DB credentials.
  # If using Amazon RDS or similar managed database services, `SWELL_DB_URL` will have the form 
  # SWELL_DB_URL=mysql://db1.abcdefghijkl.us-east-1.rds.amazonaws.com:3306/swell or similar
  SWELL_DB_URL: mysql://<DB_AND_PORT>/swell
  SWELL_DB_USER: swell
  SWELL_DB_PASSWORD: <YOUR DATABASE PASSWORD>
  SWELL_DB_DIALECT: mysql
---
apiVersion: v1
kind: Service
metadata:
  name: groundswell
  labels:
    app: groundswell
spec:
  selector:
    app: groundswell
  ports:
    - name: http
      port: 8090
      # targetPort must match with the SWELL_SERVER_PORT in the ConfigMap above.
      targetPort: 8090
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: groundswell
  labels:
    app: groundswell
spec:
  selector:
    matchLabels:
      app: groundswell
  strategy:
    type: Recreate

  template:
    metadata:
      labels:
        app: groundswell
    spec:
      imagePullSecrets:
        - name: "cr.seqera.io"
      securityContext:
        fsGroup: 101

      initContainers:
        - name: wait-for-tower-db
          image: "mysql:8.0"
          command:
            - 'bash'
            - '-c'
            - |
              echo "$(date): starting check for db $TOWER_DB_URL"
              # Strip initial Java connection string and options after '?' from URI
              until mysqlsh --uri "$(echo $TOWER_DB_URL |sed -e 's@jdbc:\(.*\)?.*@\1@')" -u"$TOWER_DB_USER" -p"$TOWER_DB_PASSWORD" --sql -e "SELECT VERSION()"; do
                echo "$(date): see you in $SLEEP_PERIOD_SECONDS seconds"
                sleep $SLEEP_PERIOD_SECONDS
              done
              echo "$(date): db server ready"
          env:
            - name: SLEEP_PERIOD_SECONDS
              value: "5"
          envFrom:
            - configMapRef:
                name: tower-backend-cfg

        - name: wait-for-swell-db
          image: "mysql:8.0"
          command:
            - 'bash'
            - '-c'
            - |
              echo "$(date): starting check for db $SWELL_DB_URL"
              # Strip initial Java connection string and options after '?' from URI 
              until mysqlsh --uri "$SWELL_DB_URL" -u"$SWELL_DB_USER" -p"$SWELL_DB_PASSWORD" --sql -e "SELECT VERSION()"; do
                echo "$(date): see you in $SLEEP_PERIOD_SECONDS seconds"
                sleep $SLEEP_PERIOD_SECONDS
              done
              echo "$(date): db server ready"
          env:
            - name: SLEEP_PERIOD_SECONDS
              value: "5"
          envFrom:
            - configMapRef:
                name: tower-groundswell-cfg

        - name: migrate-db
          image: "cr.seqera.io/private/nf-tower-enterprise/groundswell:0.3.3"
          command: ['/opt/groundswell/bin/migrate-db.sh']
          envFrom:
            - configMapRef:
                name: tower-groundswell-cfg
            - configMapRef:
                name: tower-backend-cfg

      containers:
        - name: groundswell
          image: "cr.seqera.io/private/nf-tower-enterprise/groundswell:0.3.3"
          env:
            - name: MPLCONFIGDIR
              value: "/tmp"
          envFrom:
            - configMapRef:
                name: tower-backend-cfg
            - configMapRef:
                name: tower-groundswell-cfg
          volumeMounts:
            - name: tmp-volume
              mountPath: /tmp/
          securityContext:
            capabilities:
              drop:
              - ALL
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 101

      volumes:
        - name: tmp-volume
          emptyDir:
            sizeLimit: "1Gi"

To run the service from a custom URL, declare the URL with the GROUNDSWELL_SERVER_URL environment variable in the configmap.yml file that you downloaded for your Platform installation. A non-zero value for this environment variable activates the optimization service automatically, so TOWER_ENABLE_GROUNDSWELL does not need to be set when you declare a custom URL.
Define a set of credentials for the optimization database. This can be the same database used for Seqera, but in a different schema.

If you use Amazon RDS or other managed database services, run the following commands in your database instance:

CREATE DATABASE IF NOT EXISTS `swell`;
CREATE USER 'swell'@'%' IDENTIFIED BY 'swell';
GRANT ALL PRIVILEGES ON `%`.* TO 'swell'@'%';
FLUSH PRIVILEGES;

If you do not use a managed database service, run the following commands in your database instance:

CREATE DATABASE IF NOT EXISTS `swell`;
CREATE USER 'swell'@'%' IDENTIFIED BY 'swell';
GRANT ALL PRIVILEGES ON *.* TO 'swell'@'%';
FLUSH PRIVILEGES;

The initContainers process will wait until both the Seqera and pipeline resource optimization service databases are ready before starting the migration in the Seqera database and finally starting the optimization container.

When the pipeline resource optimization service is active, pipelines that can be optimized display a lightbulb icon in your Launchpad. Any pipeline with at least one successful run can be optimized.

Docker Compose deployment​

New installation​

Existing installation​

Kubernetes deployment​

Docker Compose deployment

New installation

Existing installation

Kubernetes deployment