Analytics & Metrics

Gaffer derives four signals from the reports you upload: a health score, a flaky-test list, a pass-rate trend, and failure clusters. Each is a deterministic function of your run history. This page gives the formula, the threshold, and a worked example for every one of them.

One number to calibrate against first: a suite with a 100% pass rate, zero flaky tests, and a flat trend scores 95, not 100. Gaffer’s own dashboard suite sits exactly there. Against this repository’s local run history, which spans 2026-05-09 to 2026-08-08 and holds 2,537 tests in its newest run, gaffer query health returns a score of 95 with label excellent and trend stable, and gaffer query flaky returns an empty list. The missing 5 points are the trend bonus, which only a measurably improving suite collects.

Threshold reference

Every constant the analytics pipeline uses, in one place.

Threshold	Value	Configurable	Where it applies
Health label boundaries	90 / 75 / 50 / 25	No	Score to label mapping
Trend sensitivity	±2 percentage points	No	Improving vs stable vs declining
Flaky flip rate	10%	Yes (1-100)	Flagging a test as flaky
Flaky failure-rate band	10% to 90%	Lower bound only	Dashboard flaky detection
Minimum sample size	5 executions	No	Both flaky algorithms
Distinct failure patterns	2	No	CLI variance-based flaky detection
Cluster similarity	0.7	No	Grouping failures by error message
Analysis window	30 days	Per plan, via API	Trends, flaky detection, duration stats
Recompute interval	4 hours	Manual override	Snapshot and alert evaluation
Alert cooldown	24 hours per organization, per type	No	Health and flaky notifications

What is the Gaffer health score and how is it calculated?

The health score is a 0-100 number combining three inputs with fixed weights: pass rate at 60%, stability at 30%, and trend direction at 10%.

health = passRate * 0.6 + (100 - flakyPercentage) * 0.3 + trendBonus * 0.1

The result is clamped to 0-100 and rounded to an integer.

Term	Weight	Range	How it is derived
passRate	60%	0-100	Passed tests over passed plus failed, across the window
100 - flakyPercentage	30%	0-100	flakyPercentage is flaky tests over unique test names
trendBonus	10%	0, 50, or 100	100 improving, 50 stable, 0 declining

Pass rate carries the most weight because it is the most direct answer to “are my tests passing”. Flakiness carries 30% because a suite full of coin-flip tests is untrustworthy even at a high pass rate. Trend carries the remaining 10% as a nudge, not a verdict.

The 60/30/10 weighting is identical in the two implementations: the dashboard (server/utils/health-score.ts), which the public API’s /health endpoint imports directly, and the CLI’s local intelligence store (gaffer-core, intel/health.rs). The inputs are not identical, so the two can disagree. The CLI derives its trend by comparing the current score against the previously stored score rather than splitting a 30-day window in half, and it divides flaky tests by the total test count where the dashboard divides by the number of unique test names.

Worked examples

Every row below is an assertion in the health-score test suite, so these are the exact values Gaffer produces.

Pass rate	Flaky %	Trend	Arithmetic	Score	Label
100%	0%	stable	60 + 30 + 5	95	Excellent
100%	0%	improving	60 + 30 + 10	100	Excellent
90%	50%	stable	54 + 15 + 5	74	Needs Attention
50%	0%	declining	30 + 30 + 0	60	Needs Attention
0%	0%	stable	0 + 30 + 5	35	At Risk

The last row is worth staring at. A suite where every single test fails still scores 35, because stability and trend account for 40% of the weight and neither one collapses when the pass rate does. Read the health score next to the pass rate, never instead of it. The score is a trend instrument for a suite you already know is roughly working, not a smoke detector for a suite that just broke.

What do the health score ranges mean?

The boundaries are 90, 75, 50, and 25. The dashboard and the CLI use the same cutoffs with different words.

Score	Dashboard label	CLI label	What it should prompt
90-100	Excellent	excellent	Nothing. Watch for label transitions.
75-89	Healthy	good	Check the flaky list. At a normal pass rate, this range usually means flakiness is eating the stability term.
50-74	Needs Attention	fair	Open failure clusters. A handful of root causes is typically producing most of the failures.
25-49	At Risk	poor	The pass rate is likely below 50%. Triage failures before adding tests.
0-24	Critical	critical	Something structural broke: a bad merge, a missing service, or a config change.

How does Gaffer decide whether the trend is improving?

Gaffer buckets the last 30 days into daily pass rates, drops every day whose pass rate is not above zero, splits the remaining series in half, and compares the two averages. A gap larger than 2 percentage points sets the direction.

Difference (recent half minus older half)	Direction	Bonus
Greater than +2 points	Improving	100
Between -2 and +2 points	Stable	50
Less than -2 points	Declining	0

A worked case. Over a 30-day window you have runs on 10 days, with daily pass rates of 88, 90, 91, 89, 92 for the first five and 95, 96, 94, 97, 96 for the last five. First-half average is 90, second-half average is 95.6, difference is +5.6, so the trend is improving and the score collects the full 10 points. Days with no runs never enter the calculation, so a quiet weekend does not register as a decline. Neither does a day on which nothing passed: a 0% bucket is dropped by the same filter, so a total outage is currently invisible to the trend.

How does Gaffer detect flaky tests?

A test is flaky when it produces different outcomes for the same code. Gaffer runs two detection algorithms against two different data stores, and they answer slightly different questions.

	Dashboard and public API	CLI (gaffer query flaky)
Data store	Cloudflare Analytics Engine, SQL aggregate	Local SQLite run history
Primary signal	Failure rate inside a band	Flip rate between consecutive runs
Flag condition	10% < failure rate < 90%	flip rate ≥ 10% or ≥ 2 distinct failure patterns
Minimum sample	5 executions	5 executions
Window	30 days (plan-clamped via API)	Whatever is in the local store
Third signal	Duration variability	Distinct error-message count

Both produce a composite score from 0 to 1 and sort the list by it, so the noisiest test is always at the top.

What is a flip rate?

A flip is a change of outcome between two consecutive runs of the same test. The flip rate is flips divided by transitions.

flip_rate = flips / (executions - 1)

Given the sequence pass, fail, pass, pass, fail over five runs, there are four transitions and three of them flip (pass to fail, fail to pass, pass to fail). Flip rate is 3/4, or 75%. Skipped results are dropped from the sequence before counting, so a conditionally-skipped test does not manufacture flips.

Why does the dashboard use a failure-rate band instead of a flip rate?

Analytics Engine aggregates rows with SQL and does not preserve the per-test ordering that flip counting requires. So the dashboard asks a related question that a GROUP BY can answer: over the window, what fraction of this test’s executions failed? Tests landing strictly between 10% and 90% are neither reliably passing nor reliably failing, which is the same population flip rate is trying to isolate.

The two thresholds carve out the cases that are not flakiness:

• At or below 10% failure: an occasional real failure, or noise below your tolerance.
• At or above 90% failure: a broken test. It needs fixing, but it is not flaky, and putting it on the flaky list buries the tests that are.

How does Gaffer catch a test that fails every run?

The CLI adds a second, independent trigger: two or more distinct failure patterns. Error messages are normalized (UUIDs, timestamps, hex addresses, line numbers, and long numeric IDs are replaced with placeholders) and then counted as a set. A test that fails on all six runs has a flip rate of zero and would be invisible to flip-rate analysis, but if those six failures carry two different normalized messages, the CLI flags it anyway. Varying errors point at environmental dependence, not at one deterministic bug.

How is the composite flakiness score computed?

Both implementations weight three terms, and both cap the result at 1.0.

Dashboard:

score = 0.4 * flipProximity + 0.4 * failureRate + 0.2 * durationVariability

flipProximity is 1 - 2 * |failureRate - 0.5|, which peaks at 1.0 for a perfect 50/50 coin flip. durationVariability is (max - min) / avg, capped at 1.0.

Worked example. A test fails 8 of 20 runs, with durations ranging from 80 ms to 400 ms and averaging 150 ms.

• flipProximity = 1 - 2 × |0.4 - 0.5| = 0.8
• failureRate = 0.4
• durationVariability = (400 - 80) / 150 = 2.13, capped to 1.0
• score = 0.4 × 0.8 + 0.4 × 0.4 + 0.2 × 1.0 = 0.68

CLI:

score = 0.4 * flipRate + 0.4 * failureRate + 0.2 * patternVariance

patternVariance is (distinctPatterns - 1) / 4, capped at 1.0, so it saturates at five distinct error messages and contributes nothing when there is one or zero. A test that fails on all 5 runs with 2 distinct errors scores 0.4 × 0 + 0.4 × 1.0 + 0.2 × 0.25 = 0.45.

What should you do about a flaky test?

There is a long-running argument about whether a race condition, a shared-fixture leak, and a genuinely intermittent production bug all deserve the same word. For deciding what to do next, the distinction does not help: the cost to the team is identical. Gaffer flags the test and leaves the call to you.

Composite score	Reading	Reasonable action
0.7 and above	Near coin-flip, often slow and variable too	Quarantine now, investigate after
0.4 to 0.7	Real intermittency, moderate failure rate	Fix within the sprint
0.1 to 0.4	Marginal, or only just over the threshold	Watch. Re-check next week.

Can I change the flaky test threshold?

Yes. Settings then Analytics exposes Flip rate threshold as an integer from 1 to 100, defaulting to 10. It is organization-wide and takes effect at the next computation. Lower it to surface marginal tests, raise it if a legitimately noisy suite is drowning you in flags. The 5-execution minimum sample size is not configurable.

How is pass rate calculated?

Pass rate for the project overview is passed tests over executed tests. Skipped, pending, and todo results are excluded from both the numerator and the denominator, because a test that never ran is not evidence either way. The daily series that feeds the trend chart uses a wider denominator, described below.

pass_rate = passed / (passed + failed)

That formula runs over each run’s passed and failed counts. The daily series does not use it: it divides passed executions by every execution recorded that day, with no status filter, so skipped executions do sit in the denominator of each daily bucket. A period with a lot of skips reads lower in the trend than the overview pass rate for the same period.

Aggregation happens per calendar day, not per run, so a day with 40 runs does not outweigh a day with 2 when the trend is computed. The daily series feeds three consumers: the trend chart, the trend direction that supplies the health score’s 10% term, and the daily snapshot row that backs the historical view.

How does failure clustering work?

Clustering groups failed tests whose error messages are at least 70% similar after normalization, so a single root cause shows up as one entry instead of thirty.

The pipeline runs in three steps.

1. Normalize. Volatile substrings are replaced so that two instances of the same error compare as equal.

   Pattern	Replaced with
   UUIDs	<uuid>
   ISO 8601 timestamps	<timestamp>
   File paths with line numbers	file.ts:<line>
   id=123, id: 456	id=<id>
   Standalone numbers of 4+ digits	<id>
   Hex addresses (0x...)	<address>

2. Compare. Normalized messages are truncated to 500 characters and compared with Levenshtein distance, converted to a similarity score of 1 - distance / maxLength. Greedy assignment: the first unassigned failure seeds a cluster and every unassigned failure at or above 0.7 similarity joins it.

3. Group the remainder. Failures with no error message at all are grouped by file path instead, so they stay visible rather than collapsing into one meaningless bucket.

Clusters are returned sorted by size. The largest cluster is the highest-leverage fix on the page: 30 tests failing on one Connection refused is one problem, not thirty.

What are the failure categories?

When Workers AI is available, each cluster is also classified into one of three root-cause categories, with a confidence level and a one-sentence rationale. Results are cached for 24 hours.

Category	Means	Typical evidence
product_bug	Real application defect	Assertion failures on expected behavior, wrong return values, unexpected exceptions from app code
automation_bug	Test infrastructure issue	Stale selectors, element not found, setup and teardown failures, mock misconfiguration
environment_issue	Infrastructure problem	Connection refused, DNS failures, out of memory, certificate errors, timeouts across many tests

A cluster the model cannot place with at least medium confidence is returned uncategorized rather than guessed at.

When do analytics update?

Analytics recompute every four hours, on a cron at 00:00, 04:00, 08:00, 12:00, 16:00, and 20:00 UTC. Each cycle analyzes the trailing 30 days and does four things: recomputes the overview, writes a daily snapshot row, refreshes the cache the dashboard reads, and evaluates health alerts.

Reports uploaded between cycles are stored immediately and appear in the analytics on the next run. To skip the wait, press Compute now in Settings then Analytics. That is the right move after backfilling historical reports, where waiting four hours to find out whether the import produced sensible numbers is a bad trade.

How do health alerts work?

Gaffer sends an alert when a metric crosses a boundary, not when it moves. Two triggers exist, and each carries an independent 24-hour cooldown. Both the score and the cooldown are organization-wide, since analytics are computed across all projects in the organization.

Alert	Fires when	Payload
Health degradation	The score crosses into a worse label band	Previous score and label, current score and label
New flaky tests	Tests are detected as flaky for the first time	The list of newly flaky test names

Boundary crossing is the important detail. A drop from 94 to 91 sends nothing, because both are Excellent. A drop from 91 to 88 sends an alert, because the label changed from Excellent to Healthy. Improvements never alert.

The cooldown is per alert type, so a suite deteriorating through several bands in one afternoon produces one notification rather than four. Alerts route to your configured destinations: Slack and webhooks, set up under Settings then Notifications.

Reading the same metrics from the CLI and API

Everything on this page is available without opening the dashboard.

# Health score, label, trend, and previous score
gaffer query health

# Flaky tests, ranked by composite score
gaffer query flaky

# Duration percentiles and the slowest tests
gaffer query slowest --limit 10

The same numbers come back from the public API:

curl -H "X-API-Key: gfr_your_project_token" \
  "https://app.gaffer.sh/api/v1/user/projects/PROJECT_ID/health?days=30"

The days parameter is clamped by plan tier (Free 30, Pro 90, Team 180). When clamping happens, the response carries a meta object reporting appliedDays and requestedDays, so read the applied value rather than assuming your request was honored.

Frequently asked questions

Why does a perfect test suite score 95 instead of 100?

The trend term contributes 10 points, and a stable trend earns half of them. A suite with a 100% pass rate, zero flaky tests, and a flat trend scores 100 * 0.6 + 100 * 0.3 + 50 * 0.1 = 95. Only a measurably improving pass rate, meaning the recent half of the window averages more than 2 points above the older half, collects the full 10 and reaches 100.

How many runs does Gaffer need before it flags a test as flaky?

Five executions of the same test name. Below that, Gaffer records the results but will not classify the test, because a single failure in two runs is indistinguishable from a real bug. The minimum sample size is 5 in both the dashboard and the CLI.

Next steps

Analytics need consistent data flowing in before the trend terms mean anything. If reports are not yet uploading on every run:

• Set up the GitHub Action to upload from CI automatically
• Integrate the Upload API into a custom pipeline
• Query the metrics programmatically or through the MCP server