k6 soak test: how long to run it and what to watch

A soak test runs a steady, realistic load for a long time to find problems that only appear with age. Memory leaks, connection pools that never release, disks that fill with logs, caches that never evict. In k6 it is a ramp up, a long flat hold at normal traffic, and a ramp down.

The odd part is that k6 will often report a soak test as passing while the system is dying, because latency stays fine until the moment it does not. So most of the work is watching what k6 does not measure. This post uses a tiny service with a deliberate leak to show what that looks like. For how soak compares with load, stress and spike, see load testing types with k6 examples. For the short, sharp cousin, see the k6 spike test example.

What a soak test finds

• Memory growth that never levels off.
• File descriptors, sockets or database connections that are opened and not closed.
• Thread or worker pools that fill up slowly.
• Queues that grow because consumers are a little slower than producers.
• Disk usage from logs, temp files or uploads.
• Scheduled jobs (cron, cache refresh, token renewal) that misbehave only at the hour mark.
• Slow performance drift, such as an index that gets bigger or a table that stops fitting in memory.

None of these show up in a ten-minute load test. A system with a leak of 50 KB per request looks perfect for a long time.

Soak vs load

How long to run a soak test

There is no standard number. A better approach is to base the duration on the longest cycle in your system:

• Long enough for several garbage collection cycles under steady load, so a real trend separates from the sawtooth.
• Longer than your longest timeout or TTL. If connections idle out after 30 minutes, a 20-minute test never sees an idle connection die.
• Long enough to span your scheduled jobs at least twice. A job that runs hourly needs a run of a few hours.
• Long enough to fill whatever fills slowly. If logs rotate daily, a four-hour run sees no rotation.

As a starting point I would try four hours for a first soak, then run it overnight once the pipeline is stable. That is a heuristic, not a rule. If the trend is flat after one hour on a small service, a longer run mostly costs money. If memory is still climbing at the end, you did not run long enough to see where it settles.

The script

This is the k6 side. It ramps to twenty VUs, holds, and ramps down. The hold duration comes from an environment variable so you can run the same file for five minutes on a laptop or for hours on a CI runner.

import http from 'k6/http';
import { check, sleep } from 'k6';

const BASE = __ENV.BASE_URL || 'http://localhost:3917';

export const options = {
  scenarios: {
    soak: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: __ENV.RAMP || '5m', target: 20 }, // ramp up
        { duration: __ENV.SOAK_DURATION || '4h', target: 20 }, // hold
        { duration: __ENV.RAMP || '5m', target: 0 }, // ramp down
      ],
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<500', 'p(99)<1000'],
  },
};

export default function () {
  const res = http.get(`${BASE}/`);
  check(res, { 'status is 200': (r) => r.status === 200 });
  sleep(1);
}

Keep the load constant and realistic. A soak test at peak load is a stress test that takes longer. The goal is to hold normal traffic and watch what time does to the system.

A target that leaks on purpose

To show what a leak looks like, here is a Node service that keeps 50 KB of every request in an array and never lets go. It also exposes /metrics so we can read its memory.

const http = require('http');

const cache = []; // never evicted: the deliberate leak

http
  .createServer((req, res) => {
    if (req.url === '/metrics') {
      const { heapUsed, rss } = process.memoryUsage();
      res.setHeader('content-type', 'application/json');
      return res.end(
        JSON.stringify({
          heapUsedMB: Math.round(heapUsed / 1048576),
          rssMB: Math.round(rss / 1048576),
          cacheEntries: cache.length,
        })
      );
    }
    cache.push(Buffer.alloc(50 * 1024, 'x')); // keeps 50 KB per request
    res.end('ok');
  })
  .listen(3917);

I could not run four hours for this post, so I ran a compressed version: five-second ramps and a 50-second hold at twenty VUs, with k6 in Docker pointed at the service on my machine, while polling /metrics every ten seconds. This is a demonstration of the shape, not a soak.

And k6 said:

✓ 'p(95)<500' p(95)=2.21ms
✓ 'p(99)<1000' p(99)=2.88ms
✓ 'rate<0.01' rate=0.00%
http_reqs......................: 1110   18.28272/s

Every threshold passed. Latency was a couple of milliseconds. Meanwhile resident memory climbed from 59 MB to 127 MB, in a straight line, with the entry count rising in step. That straight line is what a leak looks like, and k6’s summary did not show it.

There is a second lesson in the table. heapUsedMB stayed flat at 5. In Node, Buffers live outside the V8 heap, so a dashboard that only graphs heap would say the service is healthy. RSS caught it. Whatever your runtime, graph the number the operating system sees as well as the number the runtime reports.

Leak or noise

Memory in a healthy service is not flat. Garbage-collected runtimes show a sawtooth: it rises, a collection drops it, it rises again. Caches fill to a limit and then stay. A real leak differs in one way. The floor of the sawtooth keeps rising.

Practical checks:

1. Compare the low points, not the average. If the minimum after each collection goes up hour after hour, that is a leak.
2. Look for a plateau. Bounded growth, such as a cache that fills to its limit, levels off. A leak does not.
3. Correlate with request count. In the table, cache entries and RSS rise together. If memory rises while traffic is flat, look at background jobs.
4. Stop sending traffic at the end and watch. A leak does not recover when the load stops. A busy pool does.
5. Repeat the run. One run can be explained by a one-off event. Two runs with the same slope cannot.

What to watch while it runs

k6 can stream its metrics to an external backend with --out, which gives you latency over time next to system metrics on the same dashboard. A summary printed at the end of a four-hour run says very little about hour three.

Mistakes

• Trusting the pass. As above, green thresholds do not mean healthy.
• Running in an environment that restarts. If your orchestrator kills and restarts a container on a memory limit, the leak is hidden. Turn that off for the test, or watch restart counts.
• Auto-scaling hiding the problem. Same idea. Pin the instance count for the test.
• Too many tiny runs. Four 30-minute runs prove less than one four-hour run.
• Forgetting the data. A soak test that writes to the database fills it. Clean up, or use a throwaway one.

A first step

Start your service locally, run SOAK_DURATION=30m k6 run soak.js, and poll your process memory every minute into a file. Plot the minimums. If the line is flat, schedule the four-hour version. If it climbs, you have found something before your users did.