Automated Regression Testing: How to Do It and Track It

Setting up automated regression testing is a solved problem: pick a runner, write the tests, wire them to a GitHub Actions job that runs on every pull request. Nearly every guide on this topic stops exactly there, at the moment the tests execute.

That is about half the job. The other half is what the suite produces: whether the pass rate is drifting down across the last month, which regression tests fail intermittently, and whether last Tuesday’s green run was green on the first attempt or the second. A regression suite whose history you cannot read is one you will eventually stop trusting, and a suite people stop trusting is one they start skipping.

What is automated regression testing?

Automated regression testing is the practice of re-running an existing body of tests, without human intervention, to confirm that code which used to work still works after a change. The tests are not new. Their value comes entirely from the fact that they passed before, so a failure now points at something the change broke.

The suite grows by accretion. A bug ships, someone fixes it, and a test goes in to make sure that specific defect never returns. Repeat for two years and the regression suite is a written record of everything the product has gotten wrong, permanently guarded.

What is automated regression?

Regression names the defect, not the technique. A regression is behavior that used to work and no longer does, and automated regression is the software-driven re-execution of already-passing tests that catches those defects after each change.

Both usages sit side by side in the same conversation. “We shipped a regression” means something that worked last week is broken this week. “We ran regression” means the suite that would have caught it was executed.

What is the difference between manual and automated regression testing?

Manual regression testing means a person re-executes test cases by hand against a checklist. Automated regression testing means a runner executes those same checks on a trigger, usually a commit or a pull request. The practical difference is cost per run: a manual pass costs hours of someone’s time, so it happens before releases, while an automated pass costs minutes of compute, so it can happen on every change.

Automation solves the execution cost. It does not solve the record-keeping problem, which is why the last row of that table reads the same in both columns: most pipelines throw the report away as soon as the job turns green.

Why automate regression testing

Automate regression testing because the frequency you need is impossible to sustain by hand. A regression suite is only useful if it runs often enough to identify which change broke something, and that means running it on every pull request rather than every release.

Gaffer’s own regression suite recorded 67,325 individual test executions in the nine days from 31 July to 8 August 2026, with 18 failures, for a 99.97% pass rate. Those executions came from the 27 test-executing runs among 41 recorded runs; the other 14 were CLI invocations that ran nothing. The largest single run covered 2,537 tests.

Nobody re-runs a 2,537-test suite by hand 27 times in nine days. That is the entire argument for automation. It is also the setup for the second half of this post: any single one of those runs is readable in a CI log, but the shape of all 27 together is not, and the interesting questions live in the shape.

Gaffer flagged zero flaky tests across those 27 executing runs between 31 July and 8 August 2026. That claim is only worth something because every run was recorded. An untracked suite cannot make the claim or refute it, which is the quiet problem with treating “the build is green” as the end of the story.

Three other things automation buys you:

• Blame localization. Running per commit means the failing change is the change under test, not one of forty in a release candidate.
• Willingness to refactor. Teams touch code they are afraid of only when something will tell them within ten minutes that they broke it.
• A history. Every run leaves a report. Whether you keep those reports is a separate decision, covered below.

How to automate regression testing

Five steps, and the first two decide whether the other three are worth doing: which tests go into the regression set, and whether those tests are deterministic enough to believe. Steps three through five are wiring.

1. Select the regression set. Start from your bug history, not from a coverage target. See the section below.
2. Make the tests deterministic and independent. Each test sets up its own data, asserts on a condition rather than a timer, and passes when run alone or in any order. Non-deterministic tests cost more than the time they waste, because each one lowers the credibility of every other result in the run.
3. Run the suite in CI on every pull request. A regression suite that only runs on a developer’s laptop protects only that developer.
4. Publish the report somewhere durable. Your runner already produces JUnit XML, a JSON summary, or an HTML report. Upload it into something you can query. A zipped artifact answers one question at a time, and only until it expires.
5. Read the trend, not the run. Any single run is a point estimate. The signal is in the shape of the last thirty.

How do you decide which regression tests to automate?

Automate the checks that guard behavior which has broken before, is expensive to break, and is cheap to assert. Everything else is negotiable. The single best input is your own bug history: if a defect shipped once, the path it took is worth a permanent test.

Prioritize:

• Paths with a defect history. Anything that has broken is likely to break again. This is the highest-yield criterion by a wide margin.
• Revenue and auth paths. Checkout, login, permissions, billing. The blast radius justifies a slow, expensive end-to-end test.
• Stable interfaces. API contracts and pure functions change slowly, so tests against them stay valuable for years.
• Anything a human is currently re-checking every release. If someone is clicking through it by hand each time, that is an automation candidate that has already justified itself.

Deprioritize:

• Interfaces still being designed. A test written against a screen that will be redesigned next sprint is maintenance debt with a short fuse.
• Behavior that is cheap to verify at a lower level. If a unit test catches the same defect in 40 milliseconds, do not write the browser test.
• Aesthetics. Automated assertions on visual polish produce more false alarms than caught bugs unless you have real visual diffing in place.

Regression testing in CI/CD

In CI/CD, a regression suite is a pipeline stage that runs on every change and blocks the merge when it fails. The configuration is short. What happens to the report afterward is the part most pipelines never configure at all.

name: Regression

on: [pull_request]

jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx playwright test --reporter=html

      - uses: gaffer-sh/gaffer-uploader@v2
        if: always()
        with:
          gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }}
          report_path: ./playwright-report

Two details in there carry more weight than they look like they do.

--reporter=html is what produces the playwright-report/ directory. Playwright’s default reporter writes to the terminal and leaves nothing on disk (list locally, dot in CI), so a workflow that skips this flag gives the upload step nothing to find. If your playwright.config.ts already sets reporter: 'html', the flag is redundant and you can drop it.

if: always() is what keeps failing runs. Without it the upload step is skipped whenever the test step fails, which means you keep a record of every run except the ones you needed to investigate. See the GitHub Actions guide for the full workflow reference.

Two structural decisions are worth making early:

• Split the suite by budget. A fast subset gates pull requests, a full run happens nightly. Ten minutes is the rough limit before developers context-switch and stop reading the output, which is the same faster CI feedback loop problem in miniature.
• Shard across runners. Most runners support splitting by worker index. This is usually a cheaper win than optimizing individual tests.

Common challenges (and how to avoid them)

Four failure modes account for most abandoned regression suites, and only one of them is about writing tests.

The suite gets too slow. Suites grow monotonically and nobody deletes tests. Shard across parallel runners, move assertions down to the cheapest layer that can catch the defect, and split the pull request set from the nightly set.

Tests become flaky and the team learns to retry. This is the most damaging one, because the failure mode is social rather than technical. Once “just hit rerun” becomes reflex, real regressions get retried away with the noise. Flaky test detection covers the diagnostic patterns, and we wrote up the practical version for browser suites in managing flaky E2E tests.

Failure volume outruns triage. A suite of 1,000 tests can produce 80 failures from a single downed dependency, and reading 80 stack traces to find one root cause is a morning nobody has. The technique is to group failures by what their error messages have in common and investigate one representative per group. Failure clustering does this automatically on every uploaded run; why the failure count isn’t the bug count walks through doing it by hand.

The results evaporate. CI artifact retention has a limit, and when it lapses the artifact is a zip you can no longer search. GitHub Actions defaults to 90 days and many teams shorten it to save storage. The moment the artifact expires, the run record becomes a green checkmark with nothing behind it.

Tracking regression results over time

Tracking regression results means keeping a durable, queryable record of what each run produced, so you can answer questions about the suite rather than about a single build. This is where suites quietly rot: without a readable history, a suite can decay for months while each individual run still reports green. Our test reporting guide makes the same argument in more depth: a report that only exists for the duration of one CI job isn’t a record, it’s a receipt.

The questions a CI run page cannot answer:

• Is the pass rate trending down across the last month, or was today just a bad build?
• Which specific tests have failed at least once in the last thirty runs without ever failing twice in a row?
• Is the suite getting slower, and which tests are responsible?
• Are today’s 40 failures 40 bugs or one broken dependency?
• Did the flaky test we quarantined three weeks ago actually get fixed?

All but one of those need run history, and the exception needs today’s failures grouped by cause. A CI run page gives you neither: it shows the current build and, at best, a list of recent builds with a colored dot next to each.

The mechanism is unglamorous. Your test runner already writes a report file. Upload it on every run, including the failing ones, into something that indexes the results instead of filing them as a zip. That is the difference that matters, and it is queryability rather than duration: a CI artifact is a file you download and read by hand, while an indexed history turns the questions above into queries instead of archaeology.

Spotting flaky regression tests before they erode trust in the suite

A flaky regression test is one that passes and fails against the same commit, and you can only identify it by comparing results across runs. A single run cannot tell you a test is flaky, because from inside one run a flaky failure and a real failure look identical.

The detection is statistical rather than clever: for each test, look at its outcomes across the last N runs and flag the ones whose results changed without the code under them changing. That is why the tracking has to exist first. A team without run history discovers flakiness socially, when someone says “oh, that one always fails, just rerun it,” which is roughly two months after the suite stopped being trustworthy.

Gaffer flags flaky tests automatically across uploaded runs, which is what makes the zero above a measurement rather than an assumption. The flaky test detection page covers the scoring.

When a regression test passes on retry, does it count as a pass?

The run passes. The test does not. A test that fails and then passes on retry against the same commit is non-deterministic by definition, and recording it as a clean pass discards the only evidence that it is flaky.

The practical resolution is to separate the two purposes. Retries are a merge gate, and it is reasonable to let a pull request through on a second attempt rather than blocking a developer on a known-racy test. First-attempt results are the health signal, and they should be recorded separately and read weekly. If your pipeline reports only the final outcome, you have configured the suite to hide its own decay.

Arguments about retries rarely resolve because each side is right about its own half and neither is arguing about the other’s.

Sharing regression health with a small team

Everyone on the team should be able to see the suite’s current state without asking the person who wrote the tests. For a small team that usually means a URL, not a login-gated enterprise report.

The practical requirements are modest: a page showing pass rate over time, the current failures grouped by cause, and which tests are flagged flaky, at a link you can paste into Slack. Gaffer’s test results dashboard does this, with Slack and webhook notifications you can set to fire on every run, on failures only, or after a number of consecutive failures you choose. Pricing is flat rather than per seat and every plan includes unlimited users, which matters mostly because per-seat pricing is the reason small teams end up with one person who checks the test results and four who do not.

Tools for automated regression testing

There is no single “regression testing tool.” A working regression automation setup is three layers, and most teams already have the first two.

The first layer is where the framework debates live, and for regression specifically the debate is moot: any of them will re-run tests on a trigger. Pick the one that matches your stack.

The third layer is the one teams skip, usually because the available options were built for enterprise buyers with test-management requirements a regression suite does not have. Gaffer is the small-team version: point the uploader at the report your runner already produces, and you get hosted reports, pass rate trends, flaky detection, failure clustering, and an MCP server so a coding agent can query the same history. Current plans and limits are listed under pricing.

Whichever tool fills that third layer, wire it up before you have a question for it. Nothing you can install today will tell you what the suite did last month if nobody was recording it then.

Frequently asked questions

How often should an automated regression suite run?

Run a fast subset on every pull request and the full suite on a schedule at least nightly. The scheduled run earns its place by catching breakage no commit caused: an expired credential, a dependency that published a new version overnight, or a third-party API that changed behavior while nobody was pushing code.

What is the difference between regression testing and smoke testing?

A smoke test asks whether the build is functional enough to be worth testing further, and covers a handful of critical paths in a minute or two. Regression testing asks whether anything that previously worked has broken, and covers as much prior behavior as you can afford to run. Smoke is a subset in spirit, though most teams maintain it as a separate, deliberately tiny suite.

How long should an automated regression suite take?

There is no correct number, and the useful threshold is behavioral rather than numeric: a suite is too slow once developers stop reading its output and start reading only the red or green badge. You can observe that moment instead of guessing at it. If people open a new task while the suite runs, the feedback has already stopped landing, and shaving another minute off the runtime will not bring them back.

Do you need a test management platform to track regression results?

No. Test management platforms are built around planning and tracking manual test cases: writing them up, assigning them to testers, recording who executed which case against which build, and mapping cases back to requirements. An automated regression suite has no manual execution step, so most of that machinery has nothing to attach to. It needs a record of what each automated run produced, which is a far smaller problem than the one those platforms are priced to solve.