Custom Distributed Task Execution on Bitbucket Pipelines

Using Nx Agents is the easiest way to distribute task execution, but it your organization may not be able to use hosted Nx Agents. With an enterprise license, you can set up distributed task execution on your own CI provider using the recipe below.

Run Custom Agents on Bitbucket Pipelines

Run agents directly on Bitbucket Pipelines with the workflow below:

1image: node:20 2 3clone: 4 depth: full 5 6definitions: 7 steps: 8 - step: &agent 9 name: Agent 10 script: 11 - export NX_BRANCH=$BITBUCKET_PR_ID 12 13 - npm ci 14 - npx nx-cloud start-agent 15 16pipelines: 17 pull-requests: 18 '**': 19 - parallel: 20 - step: 21 name: CI 22 script: 23 - export NX_BRANCH=$BITBUCKET_PR_ID 24 - export NX_CLOUD_DISTRIBUTED_EXECUTION_AGENT_COUNT=3 25 26 - npm ci 27 - npx nx-cloud start-ci-run --stop-agents-after="e2e-ci" --agent-count=3 28 - npx nx-cloud record -- nx format:check 29 - npx nx affected --target=lint,test,build,e2e-ci --parallel=2 30 - step: *agent 31 - step: *agent 32 - step: *agent 33

This configuration is setting up two types of jobs - a main job and three agent jobs.

The main job tells Nx Cloud to use DTE and then runs normal Nx commands as if this were a single pipeline set up. Once the commands are done, it notifies Nx Cloud to stop the agent jobs.

The agent jobs set up the repo and then wait for Nx Cloud to assign them tasks.

Two Types of Parallelization

The agents and the --parallel flag both parallelize tasks, but in different ways. The way this workflow is written, there will be 3 agents running tasks and each agent will try to run 2 tasks at once. If a particular CI run only has 2 tasks, only one agent will be used.