Flaky Tests in CI: Detection and Remediation

Most developers have seen this CI problem: a test fails, then passes after a rerun, even though the code did not change.

That is a flaky test. The pull request is green again, but the failure still matters. Treat the passing retry as evidence to investigate, not proof that the problem disappeared.

A passing retry does not clear a failure

A flaky test passes and fails with the same inputs. Its result can change because of shared state, timing, test order, a port collision, or an external service.

A failure after a source or configuration change may be a regression instead. Fix a failure that reproduces with the changed inputs. Do not retry it until it passes.

Stop rerunning the whole pipeline

A full-pipeline retry repeats work that already passed. It wastes compute, delays feedback, and makes failing CI easier to ignore.

Re-run the failed test or test task by itself. A narrow retry gives you a cleaner result and keeps the rest of the pipeline moving.

Check the common causes

Most flakes match one of these patterns:

Pattern	Typical signal	Usual fix
Shared state	The test passes alone but fails after another test.	Reset data, files, mocks, and process state for each test.
Fixed ports or names	Parallel tasks collide or one task cannot start a service.	Allocate isolated ports, directories, and test data.
Timing assumptions	The failure follows a slow machine or delayed response.	Wait for a defined event or state, not an arbitrary delay.
Test-order dependence	A different order changes the result.	Make each test create and clean up its own prerequisites.
External services	Network, API, or third-party failures appear in the logs.	Stub the dependency where practical, or isolate and monitor the integration.

Manual investigation limits wasted work, but teams also need reliable evidence across CI runs. Nx provides that evidence and automates narrow retries.

Detect and retry flaky tasks with Nx Cloud

Nx runs a test command as a task. A task can run one test file, a group of files, or an entire test suite. Nx creates a hash from that task's inputs each time it runs.

When one task hash both fails and succeeds, Nx Cloud knows that task is flaky. This is stronger evidence than a retry count. A test that fails once after a new commit may be a regression. A task that passes and fails with one hash is nondeterministic.

When a known flaky task fails, Nx Cloud sends it to a different Nx Agent. It makes at most two attempts in total. The retry targets the failed work instead of the complete pipeline.

Enable flaky task detection to use this behavior in your CI pipeline.

Use flaky task analytics to choose what to fix

The Nx Cloud dashboard shows active flaky tasks, average flake rate, and high-risk tasks. It ranks tasks by impact, which combines a task's flake rate with how often it runs.

Start with a frequently run task that fails often. It blocks more pull requests and wastes more CI time than a task with one isolated failure. Open the task to inspect its attempts, logs, and execution environments before you change the test.

Fix the flaky task

Use the attempt details from Nx Cloud and work through the failed task in this order:

1. Re-run the failed test or test task by itself.
2. Compare the failed attempt with the successful attempt.
3. Check the logs, machine image, environment values, network calls, and test order.
4. Fix the cause that differs between attempts.
5. Quarantine the test only when the team cannot fix it immediately.

Record an owner and review date for each quarantine. A quarantine protects the pipeline, but it also reduces test coverage.

Make e2e retries smaller with Nx

A failed spec should not re-run a 40-minute e2e suite. Nx can split a large suite into independently runnable file-level tasks. Then Nx Cloud can retry the failed spec and leave successful specs alone.

Automated task splitting creates one task per test file for supported Nx plugins. Fix hidden shared state before you distribute a suite across workers or machines.

Keep task inputs complete

A test can appear flaky when its execution context changed but the task did not include that change in its inputs. Include configuration, environment values, generated files, and other dependencies that affect the result.

How caching works explains how Nx uses task inputs. Task sandboxing can find undeclared task dependencies.

For the wider CI design, use Monorepo CI best practices. It covers affected runs, caching, distribution, test splitting, and flaky-task handling together.