Custom Distributed Task Execution on GitLab

Using Nx Agents is the easiest way to distribute task execution, but it your organization may not be able to use hosted Nx Agents. With an enterprise license, you can set up distributed task execution on your own CI provider using the recipe below.

Run Custom Agents on GitLab

Run agents directly on GitLab with the workflow below:

1image: node:18 2 3# Creating template for DTE agents 4.dte-agent: 5 interruptible: true 6 cache: 7 key: 8 files: 9 - yarn.lock 10 paths: 11 - '.yarn-cache/' 12 script: 13 - yarn install --cache-folder .yarn-cache --prefer-offline --frozen-lockfile 14 - yarn nx-cloud start-agent 15 16# Creating template for a job running DTE (orchestrator) 17.base-pipeline: 18 interruptible: true 19 only: 20 - main 21 - merge_requests 22 cache: 23 key: 24 files: 25 - yarn.lock 26 paths: 27 - '.yarn-cache/' 28 before_script: 29 - yarn install --cache-folder .yarn-cache --prefer-offline --frozen-lockfile 30 - NX_HEAD=$CI_COMMIT_SHA 31 - NX_BASE=${CI_MERGE_REQUEST_DIFF_BASE_SHA:-$CI_COMMIT_BEFORE_SHA} 32 - NX_CLOUD_DISTRIBUTED_EXECUTION_AGENT_COUNT=3 # expected number of agents 33 34 artifacts: 35 expire_in: 5 days 36 paths: 37 - dist 38 39# Main job running DTE 40nx-dte: 41 stage: affected 42 extends: .base-pipeline 43 script: 44 - yarn nx-cloud start-ci-run --stop-agents-after=e2e-ci 45 - yarn nx-cloud record -- nx format:check --base=$NX_BASE --head=$NX_HEAD 46 - yarn nx affected --base=$NX_BASE --head=$NX_HEAD -t lint,test,build,e2e-ci --parallel=2 47 48# Create as many agents as you want 49nx-dte-agent1: 50 extends: .dte-agent 51 stage: affected 52nx-dte-agent2: 53 extends: .dte-agent 54 stage: affected 55nx-dte-agent3: 56 extends: .dte-agent 57 stage: affected 58

This configuration is setting up two types of jobs - a main job and three agent jobs.

The main job tells Nx Cloud to use DTE and then runs normal Nx commands as if this were a single pipeline set up. Once the commands are done, it notifies Nx Cloud to stop the agent jobs.

The agent jobs set up the repo and then wait for Nx Cloud to assign them tasks.

Two Types of Parallelization

The agents and the --parallel flag both parallelize tasks, but in different ways. The way this workflow is written, there will be 3 agents running tasks and each agent will try to run 2 tasks at once. If a particular CI run only has 2 tasks, only one agent will be used.