Smoke testing vs regression testing: how they differ and where each runs in CI

Smoke testing checks that a build is alive. A handful of tests walk through the most important paths, and if any of them fails, nobody spends time on the rest. Regression testing checks that what worked before still works after a change. It is a much larger suite, and it takes longer to run.

That is the whole difference. The rest of this post is how it plays out in a team: how big each suite should be, who triggers it, and where it lives in a pipeline. If you want all three terms (smoke, sanity and regression) in one place, I covered them in what is smoke, sanity, regression testing. This post only compares two of them.

Smoke testing vs regression testing in a table

The numbers in the “Size” and “Speed” rows are rules of thumb, not measurements. What matters is the ratio. Smoke has to be small enough that people never feel tempted to skip it.

Is smoke testing part of regression testing?

In most teams, yes. Smoke tests are a subset of the regression suite that you promote because they cover the highest-risk paths. You do not write them twice. You tag them.

The reverse is not true. A green regression run says a lot about your product. A green smoke run only says it is worth running the regression suite.

Some teams keep smoke tests as a separate suite that runs against a deployed environment, where regression tests would need seeded data or mocks. That works too. What goes wrong is when “smoke” quietly grows into 300 tests and stops being a fast gate.

What belongs in a smoke suite

Pick tests by cost of failure, not by feature count. For a typical web shop that could be:

• The home page returns 200 and renders.
• A user can log in.
• A user can add an item to the cart and reach checkout.
• The API health endpoint answers.

Four or five checks like these catch the bad deploys: a missing environment variable, a broken migration, a front end pointing at the wrong API. They do not catch a rounding bug in the discount code. That is regression’s job.

A useful rule is that every smoke test should be able to fail the whole release by itself. If you would shrug at one of them failing, move it to regression.

Where each runs in a pipeline

Here is the shape I use as a starting point. Smoke runs on every push to the main branch and blocks everything after it. Regression runs on a schedule, so the long run does not slow down merges.

The test suite below is Playwright, where tags go in the test title and you filter with --grep. Any runner with tags or folders works the same way.

// tests/checkout.spec.ts
import { test, expect } from '@playwright/test';

test('user can reach checkout @smoke', async ({ page }) => {
  await page.goto('/');
  await page.getByRole('button', { name: 'Add to cart' }).first().click();
  await page.getByRole('link', { name: 'Cart' }).click();
  await expect(page.getByRole('heading', { name: 'Checkout' })).toBeVisible();
});

test('discount code is rounded to two decimals', async ({ page }) => {
  // regression-only: no @smoke tag
});

And the GitHub Actions workflow with two jobs:

name: tests

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 2 * * *' # nightly regression

jobs:
  smoke:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --grep @smoke

  regression:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test

The regression job runs everything, including the smoke tests. That costs a few extra minutes and saves you from maintaining two lists. When the regression run grows past what a nightly window can take, split it across machines. Parallelisation in CI/CD covers how to do that without making the suite flaky.

Smoke tests for a deployed service

A deployed smoke test is slightly different from one in CI. It runs against the real environment right after a release, so it has to be read-only or use a dedicated test account. No creating orders for real customers.

For an API this can be a script with a few requests:

curl --fail --silent https://staging.example.com/health
curl --fail --silent https://staging.example.com/api/products | head -c 200

--fail makes curl exit non-zero on an HTTP error, so the pipeline step fails when the service is down. Wire that step to run after the deploy, and let a failure trigger a rollback or at least a page. This is the same idea as the in-pipeline check, applied one stage later.

Common mistakes with this split

• The smoke suite keeps growing. Every bug seems important enough to add. Set a budget (say, under five minutes) and remove one test for every one you add.
• Smoke tests that depend on each other. If test three needs the state from test two, a failure in two hides everything after it.
• Regression only before release. Then a bug from three weeks ago shows up on release day, and nobody remembers which change caused it. Run it nightly, even if the result is only read in the morning.
• Treating the nightly run as optional. A red nightly build that nobody owns turns into noise within a month. Give it an owner.

FAQ

Can smoke tests replace regression tests? No. Smoke tests are shallow on purpose. They tell you the build is not obviously broken, not that it is correct.

How many smoke tests should I have? Enough to cover the paths where a failure would stop the business, and few enough that the run finishes in minutes. For most apps that is a few to a few dozen.

Do smoke tests run before or after deployment? Both are valid. Before deployment they run on a preview or staging build to gate the release. After deployment they confirm production came up healthy.

A first step

Open your current suite and mark the five tests you would run if you had two minutes. Tag them @smoke, add a pipeline job that runs only those on every merge, and move the full run to a schedule. Once that works, sanity testing vs smoke testing is the next distinction worth getting straight.