Flaky end-to-end tests: the five causes and how to fix each

A flaky test passes and fails on the same code. Almost every flaky end-to-end test I have seen comes from one of five causes: a timing race, shared state, order dependence, environment limits, or an unstable selector. Name which one you have before you touch the test. Adding a retry or a longer timeout hides the cause and leaves it in the suite.

Here are the five in one table. The names stay the same through the rest of the post.

Each section below reproduces one cause in a small Playwright test and fixes it. The examples run against a small local Node server that keeps a cart and a record in memory, shown below, so you can copy everything and see the same failures. I did not measure flake rates and I won’t quote any. A cause is either present in the code or it is not.

The demo server

// server.js, run with: node server.js
const http = require('http');

const carts = {};      // userId -> items
const records = {};    // id -> name

http.createServer((req, res) => {
  const url = new URL(req.url, 'http://localhost:3100');
  const send = (code, body, type = 'application/json') => {
    res.writeHead(code, { 'content-type': type });
    res.end(typeof body === 'string' ? body : JSON.stringify(body));
  };

  if (url.pathname === '/') {
    return send(200, `<button id="load">Load</button><p id="status">Idle</p>
      <script>
        document.getElementById('load').onclick = () =>
          setTimeout(() => (document.getElementById('status').textContent = 'Done'), 300);
      </script>`, 'text/html');
  }
  if (url.pathname === '/cart' && req.method === 'POST') {
    const user = url.searchParams.get('user') || 'shared';
    (carts[user] ||= []).push('item');
    return send(200, { count: carts[user].length });
  }
  if (url.pathname === '/records' && req.method === 'POST') {
    records[1] = 'draft';
    return send(200, { id: 1 });
  }
  if (url.pathname === '/records/1' && req.method === 'PUT') {
    if (!records[1]) return send(404, { error: 'missing' });
    records[1] = 'published';
    return send(200, { status: records[1] });
  }
  send(404, {});
}).listen(3100);

The Playwright config points baseURL at http://localhost:3100, sets fullyParallel: true and workers: 4, and starts the server through webServer.

What flaky means

A flaky test has no stable result for a fixed input. The input includes more than the code: time, data, the order other tests ran, the machine. If the same commit gives you green on one run and red on the next, something in that wider input moved.

That is also why rerunning is the wrong first move. A rerun tells you the test is flaky, which you already know. It does not tell you which of the five you are looking at.

Timing race

The test reads the page before the page has finished changing.

test('bad: reads text once', async ({ page }) => {
  await page.goto('/');
  await page.click('#load');
  expect(await page.locator('#status').textContent()).toBe('Done');
});

The page updates #status 300 ms after the click. The test reads it immediately. On my machine this fails every time, because the delay is fixed. In a real app the delay varies with load, and that is how a test that always fails here becomes one that fails one run in thirty there.

The fix is to let the assertion retry:

test('good: retries until the text matches', async ({ page }) => {
  await page.goto('/');
  await page.click('#load');
  await expect(page.locator('#status')).toHaveText('Done');
});

expect(locator).toHaveText polls until it matches or the timeout expires. The same applies to any waitForTimeout(2000) in a suite. A fixed sleep is the timing race written down: too short on a bad day, wasted time on a good one. Replace it with the condition you were waiting for.

Shared state

Two tests touch the same record at the same time. Browser contexts are isolated in Playwright, so shared state usually lives behind the page: a database row, a user account, a cart on the server.

for (const n of [1, 2, 3, 4]) {
  test(`bad: shared cart ${n}`, async ({ request }) => {
    const res = await request.post('/cart');
    expect((await res.json()).count).toBe(1);
  });
}

With four workers, each test adds an item to the one cart and expects to see exactly one. In my run three of the four failed. Which ones failed changed from run to run, which is the signature of shared state: the failing test is whichever got there second.

The fix is data that belongs to the test:

test(`good: own cart ${n}`, async ({ request }, testInfo) => {
  const user = `u-${testInfo.workerIndex}-${testInfo.testId}`;
  const res = await request.post(`/cart?user=${user}`);
  expect((await res.json()).count).toBe(1);
});

In a real suite, create a user through the API in a fixture and delete it afterwards. If you cannot make data per-test, the honest option is to run that file serially with test.describe.configure({ mode: 'serial' }), and to accept that it will never parallelize. Parallel runs and what breaks in them are covered in the post on parallelisation in CI/CD and How to run tests in parallel in CI/CD.

Order dependence

Test B only passes if test A ran before it.

test.describe('bad: B needs A', () => {
  test('A creates a record', async ({ request }) => {
    expect((await request.post('/records')).ok()).toBe(true);
  });
  test('B publishes it', async ({ request }) => {
    expect((await request.put('/records/1')).ok()).toBe(true);
  });
});

The full run is green. Run npx playwright test -g "B publishes" and B fails with a 404, because A never ran. It also breaks when a runner reorders files, shards the suite, or retries only A.

The difference from shared state: here nothing runs concurrently. The suite just encodes a hidden script. The fix is for B to set up its own record:

test('publishes a record', async ({ request }) => {
  await request.post('/records');
  expect((await request.put('/records/1')).ok()).toBe(true);
});

A quick check for any suite: execute each test file alone, then each test alone with -g. Anything that fails alone is order dependent.

Environment limits

The test is correct, and the machine running it is slower than the test assumes. CI runners often have fewer CPU cores than a laptop, and they run the browser, the app and sometimes the database on the same box. Hit them with the same worker count you use locally and timeouts follow.

To reproduce it, make the dependency slow on purpose:

await page.route('**/', async (route) => {
  await new Promise((r) => setTimeout(r, 300));
  await route.continue();
});
await page.goto('/');
await page.click('#load');
await expect(page.locator('#status')).toHaveText('Done', { timeout: 100 });

The assertion budget is smaller than what the system needs, and it fails. Real cases are the same shape with bigger numbers: a default 5 s assertion timeout against an endpoint that takes 6 s on a cold CI cache.

There are three honest fixes, in this order:

1. Make the slow thing faster, or stub it, if it is not what the test is about.
2. Give that one assertion or test its own timeout. test.slow() triples the timeout for a single test and says in the code that it is slow on purpose.
3. Lower workers in CI to match the machine. A suite that runs four workers on a two-core runner is asking for this.

Do not raise the global timeout. It makes every real failure take longer to appear, and the slow test is still slow.

Unstable selector

The locator matches more than one element, or something that changes with the build.

await page.setContent(
  '<div class="btn"><button>Save</button></div><div class="btn"><button>Cancel</button></div>'
);
await page.locator('.btn button').click();

Playwright refuses with a strict mode violation, because the selector resolved to two elements. That is the friendly version. The unfriendly one is a selector that matched one element yesterday and a different one today, after a designer added a wrapper. Generated class names from CSS-in-JS tools do the same.

await page.getByRole('button', { name: 'Save' }).click();

Role and accessible name are how a user would describe the button, so they change less often than markup does. When there is no good role, add a data-testid and treat it as a contract with the front-end team. More on this in How to make your UI automation tests resilient to changes.

Tracing a flake

Guessing at causes is slow. Playwright can record the evidence for you.

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: { trace: 'on-first-retry' },
});

With this config, a test that fails records a trace on its first retry. A test that fails and then passes on retry is reported as flaky, not failed, and its trace is attached. Open it with npx playwright show-trace trace.zip, or from the HTML report. You get a timeline of actions, a DOM snapshot before and after each one, network calls and console output.

To reproduce locally, repeat the test until it breaks:

npx playwright test login.spec.ts --repeat-each=50 --workers=4

Read the trace against the table above. A click that happened before an element settled is a timing race. A request for data the test did not create is shared state or order dependence. A step that took ten times longer than the others points at the environment.

Quarantine policy

Sometimes you cannot fix a flake today. Keep it from blocking everyone, and keep it visible.

• Tag it: test('checkout', { tag: '@flaky' }, ...).
• Exclude it from the merge gate with --grep-invert @flaky, and run those tests in a separate non-blocking job.
• Give each one an owner and a date in the ticket. A quarantine with no owner is a deletion that has not been admitted yet.
• Delete the tag when the cause is fixed, and delete the test if it protects nothing.

Quarantine is not the same as retries. Retries make the suite pass. Quarantine takes a known-bad test out of the signal. I would use retries only to collect traces, and quarantine to keep the pipeline honest.

FAQ

What is the most common cause of flaky E2E tests? Timing races: the test checks the page before it finishes updating. Replacing fixed sleeps and one-time reads with retrying assertions removes most of them.

Should I add retries to fix flaky tests? Use retries to capture a trace, not to hide the failure. A test that passes on its second try is still flaky and still costs you time.

How do I find out which test is flaky? Run the suite with retries on and read the report: tests that failed then passed are marked flaky. Then run that test with --repeat-each to reproduce it.

Do flaky tests only happen in E2E suites? No, unit tests flake on time and shared state too. E2E suites suffer more because they involve a browser, a network and a backend.

Pick the five slowest or most-rerun tests in your suite, run each alone with --repeat-each=20, and label every failure with one of the five causes. The labels tell you where to spend the week.