How to run tests in parallel in CI/CD: sharding, matrices and what breaks

To run tests in parallel in CI/CD, split the suite at one of three levels: workers inside one machine, shards across several machines, or separate jobs per test group. Each level cuts wall-clock time, and each one exposes tests that quietly depend on shared state. Most of the work is fixing those tests, not writing the pipeline.

I covered why parallel runs matter in the post on CI/CD and parallelisation. This one is the how: what to configure, a GitHub Actions example you can copy, and what breaks first.

Three levels of parallelism

You can combine them. A common setup is four shards, each running two workers.

Start with workers, because they cost nothing. Add shards when one machine is no longer enough. Split by group when the groups have different purposes, as in smoke testing vs regression testing, where the fast group should gate the slow one.

Workers: the cheapest speedup

Most runners have a flag for this. Playwright uses --workers, Jest uses --maxWorkers, pytest uses the pytest-xdist plugin with -n. Check how many cores the CI runner has before picking a number. Setting workers higher than the core count usually makes browser tests slower and flakier, because the machine starts starving.

Workers run in separate processes, so anything in memory is isolated. Anything outside memory is shared: the file system, the database, the ports, the third-party account the tests log into.

A GitHub Actions matrix with sharding

Playwright has a --shard flag that takes the index and the total, for example --shard=2/4. A matrix turns that into four parallel jobs. This workflow follows the structure from the Playwright sharding docs:

name: e2e
on:
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: blob-report-${{ matrix.shardIndex }}
          path: blob-report
          retention-days: 1

Three details matter here.

fail-fast: false keeps the other shards running when one fails. With the default, GitHub cancels the siblings, and you only see the first failure. For a test suite you want the full list.

cache: npm on setup-node caches the npm download cache, so each shard does not fetch every package again. Browsers are a separate download, so they are not covered by that line.

The shard count is a number in the matrix, so changing from four to eight machines is a one-line edit. To see which count is worth it, run the suite with two, four and eight shards and compare the total time and the number of billed minutes. Billed minutes go up as you add shards, because every shard pays for checkout, install and browser setup. Wall-clock time stops improving when that fixed cost is bigger than the tests it saves.

Sharding a suite so the shards are even

Playwright shards by test file by default, and with fullyParallel: true in the config it can split individual tests. If one file holds 40 slow tests and the rest hold 3 each, file-level sharding gives you one slow shard and three idle ones. Two fixes:

• Enable fullyParallel: true so tests, not files, are the unit. This only works if tests inside a file do not depend on each other.
• Split giant files by feature. This is good hygiene anyway.

Other runners work on the same principle. Jest has --shard=1/4. For pytest, plugins and CI-native test splitting (CircleCI, GitLab) can split by recorded timings. Splitting by timing is more even than splitting by count, because a test that takes 90 seconds is not the same as one that takes 2.

What breaks when you go parallel

Passing in sequence and failing in parallel always means shared state. These are the usual suspects.

Shared test data

Two tests log in as the same user and one changes the password. Two tests create an item called “Test item” and then each assert there is exactly one. Fix it by making data unique per test: generate the user inside the test, or add a worker index or random suffix to names. Creating data through an API in a setup step is faster than building it through the UI.

Hard-coded ports and files

Two workers that start a mock server on port 3000 collide, and the second one fails with “address in use.” Let the OS choose a free port, or derive the port from the worker index. The same goes for temp files and download folders: give each worker its own directory.

Order dependence

Test B passes only because test A ran first and left a record behind. In a single process this works for months. After sharding, B lands on a different machine than A and fails. Each test should create what it needs, and clean up or ignore what is left over. Test independence is a design decision, which I wrote about in six things to consider while designing a test architecture.

Shared external state

A staging environment, a third-party sandbox or a rate-limited API takes the combined load of every shard at once. Symptoms are timeouts and 429 responses that appear only at higher shard counts. Reduce shared calls, mock the external service for most tests, or lower parallelism for the tests that touch it by putting them in their own job.

Resource starvation

More workers than cores gives timeouts that look like product bugs. Before blaming the application, drop the worker count and see if the failures go away.

If a failure shows up only when the suite runs in parallel, run the failing test alone ten times, then run it with its neighbours. A bug that needs company is shared state.

Merging reports from many shards

Four shards produce four reports, which nobody wants to read. Playwright’s blob reporter writes a compact report per shard, and a final job merges them:

  merge-reports:
    if: ${{ !cancelled() }}
    needs: [test]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
          cache: npm
      - run: npm ci
      - uses: actions/download-artifact@v4
        with:
          path: all-blob-reports
          pattern: blob-report-*
          merge-multiple: true
      - run: npx playwright merge-reports --reporter html ./all-blob-reports
      - uses: actions/upload-artifact@v4
        with:
          name: html-report
          path: playwright-report
          retention-days: 14

Set reporter: process.env.CI ? 'blob' : 'html' in playwright.config.ts so the shards write blob reports in CI and a normal HTML report on your laptop. The if: ${{ !cancelled() }} on the merge job matters: without it, the job that explains the failures is skipped exactly when there are failures.

When not to parallelize

Parallelism is not free. Skip it, or limit it, in these cases:

• The suite takes two minutes. Setup cost per shard will eat the gain.
• Tests share one stateful environment that cannot be duplicated, such as a single shared database with no reset. Fix isolation first.
• The suite is flaky in sequence. Parallel runs will multiply the noise. Make it stable at one worker, then scale.
• You pay per minute and the team is small. Four shards cost more billed minutes than one job, even though they finish sooner.

A practical next step

Run your suite today with the worker count set to 1, then to 2. If a test fails at 2, you have found your first shared-state bug, and it is cheaper to find it now than after you have built a matrix around it. Once two workers pass ten runs in a row, add the shard matrix above with two shards and check the wall-clock time.