A Python service, a TypeScript frontend, and a Rust worker in one repo means four report formats in one CI run. Downloading them is annoying. The real problem shows up when you try to answer “is the suite healthier than last week?” and discover the suites disagree about what they are counting.
Gaffer’s own CI uploads three report formats from three test suites into one project on a code pull request: a Vitest HTML report bundled with LCOV coverage, JUnit XML from the 331 gaffer-parsers test functions run under cargo-nextest, and a Playwright HTML report from the browser suite. Two of those three, the Vitest HTML report and the Rust JUnit XML, carry no retry count and no flaky flag, so any flakiness figure for those suites has to be computed from run history rather than read out of the file.
Why does one repo end up with four report formats?
Nobody picks four formats. Each framework ships its own reporter, and the reporter is the path of least resistance at the moment you add that framework.
JUnit XML became the common denominator by accident. It started as Ant’s output format for Java, has no formal specification, and every framework that emits it invented its own conventions on top. CTRF is the deliberate attempt at a shared JSON schema, but adoption is per-framework, so it arrives only where someone installed the reporter. Playwright and .NET each ship a native format that carries more than JUnit XML can express, which is a good reason to keep them and a bad reason for comparability.
So the sprawl is not a discipline problem. It is the sum of five reasonable local decisions.
What does each test report format actually carry?
The formats are not interchangeable subsets of each other. Each drops something the others keep. This is the field-level picture Gaffer’s parsers work from:
| Format | Skip representation | Retries | Flaky | File and line | Timing detail |
|---|---|---|---|---|---|
| JUnit XML | <skipped> child element | not carried | not carried | only if the writer sets file / line attributes | per-test time, seconds |
| Jest / Vitest JSON | pending, todo, skipped, disabled | not carried | not carried | file path plus line | per-test duration, suite start and end |
| Playwright JSON | skipped status | retry index per attempt | native flaky status and run-level count | spec file plus line | per-test start time and worker index |
| CTRF JSON | skipped, pending, other | retries count | per-test boolean plus summary count | optional filePath and line | per-test duration, run start and stop |
| TRX (.NET) | NotExecuted, NotRunnable, Inconclusive | not carried | not carried | not carried | per-test duration as a .NET timespan |
Two consequences fall straight out of that table. Only Playwright JSON and CTRF can report flakiness in the file itself. Only Playwright JSON and TRX distinguish a timeout from an ordinary failure, through Playwright’s timedOut result status and TRX’s Timeout outcome.
What breaks when you compare suites that disagree?
Four specific things, and all four are invisible until you put the numbers side by side.
Pass rate has a different denominator per format
Take one run of 100 tests where three failed on the first attempt and passed on retry.
- Playwright JSON puts those three in a separate flaky bucket. They are neither passed nor failed, and the run total is expected plus unexpected plus flaky plus skipped. Divide passed by total and you get 97%.
- Jest or Vitest JSON has no flaky bucket. The same three tests are simply passed, and you get 100%.
Same run, same underlying reality, three-point spread. Stack a few frameworks and the aggregate number is meaningless.
Why does the skip count differ between suites?
A skip in JUnit XML is a <skipped> child element. In Jest and Vitest JSON it is one of pending, todo, skipped, or disabled. In CTRF it is skipped, pending, or other. In TRX it is NotExecuted, NotRunnable, or Inconclusive, which lumps “we chose not to run this” together with “this could not produce a verdict”. Any tool that pattern-matches on the literal string gets a different skip count from each suite.
Retries are recorded in one format and inferred in the rest
Playwright JSON records a retry index on every attempt and CTRF records a retries count. JUnit XML, Jest/Vitest JSON, and TRX record nothing. Some JUnit generators bolt on a <rerunFailure> element, but it is not standardized, so parsing it is per-generator work. See the JUnit XML format guide for what the schema does and does not define.
Parameterized and matrix runs collapse unless the name carries the parameter
Cross-run analysis groups executions by test name. The CLI records Playwright’s full test path with the project name appended, so the same spec under chromium, firefox, and webkit stays three distinct histories. JUnit XML has no equivalent: parameterized cases are distinguishable only to the extent the generator wrote the parameter into the test name. A generator that does not collapses an entire matrix into one history, and the flakiness signal for that test becomes noise.
What do teams do about it, and why does each approach fall short?
1. Standardize every suite on JUnit XML
- Genuinely gives you one parser and one shape
- Levels down to the weakest format. Playwright’s retry data, trace links, and browser metadata are gone the moment you convert
- TRX outcome granularity collapses too:
InconclusiveandNotExecutedboth become a bare<skipped>
2. One CI job per framework, read the artifacts separately
- No conversion, no data loss
- No shared denominator, so there is no suite-wide number to trend
- Artifacts expire on different clocks, so historical comparison degrades unevenly
3. Write a merge script
- Works, right up until it is load-bearing
- You now own a parser per format, forever, and a framework upgrade can change its JSON shape without warning
- Merging summaries is the easy half. Reconciling status vocabularies is the half that gets skipped
4. Buy a framework-native tool per framework
- Deeper framework-specific coverage than any generic parser gives you
- Two or three bills, two or three dashboards, and still no cross-suite number
- Tools tied to one runner cannot answer “which layer is least reliable this month”
How does Gaffer normalize across frameworks?
Gaffer parses each format natively, then maps every test onto one of five statuses: passed, failed, skipped, timedOut, or flaky. Everything is stored against the CTRF field set, so a Rust suite and a .NET suite land in the same columns and the comparison is arithmetic rather than interpretation.
Three parts of that are worth spelling out, because they are what makes the cross-framework number trustworthy.
Skip vocabularies are folded, not string-matched. Jest’s pending and todo both become skipped. CTRF’s skipped, pending, and other are summed into the same bucket. TRX’s NotExecuted, NotRunnable, and Inconclusive follow. One skip count per run, whatever emitted it.
Flakiness is computed from history, not read from the file. Since three of the five formats cannot express flakiness at all, trusting the file would mean a JavaScript suite gets flaky detection and a Python suite does not. Instead, Gaffer groups executions by test name across runs, ignores anything that is not a pass or a fail, and requires at least five executions before judging a test. It flags a test when the pass/fail sequence flips on 10% or more of consecutive run pairs, or when the test fails with two or more distinct error messages after normalization. The second rule catches a test that fails consistently for shifting reasons, which pure flip-rate analysis misses. Details in flaky test detection.
Health is one score per project, not per runner. The score weights pass rate at 60%, stability at 30%, and trend direction at 10%, over the normalized statuses. Because the inputs are normalized, adding a fourth suite in a fourth language changes the score by the amount that suite actually contributes.
How do you get every suite into one dashboard?
Upload every report to the same project and tag each one. Reports join on commit SHA, so suites that run in different jobs, on different runners, in different languages still land under the same commit.
This is the pattern to follow:
# Job 1: Vitest unit tests, emitted as JUnit XML plus HTML and LCOV- name: Upload unit results if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: apps/dashboard/test-reports commit_sha: ${{ github.sha }} branch: ${{ github.ref_name }} test_framework: vitest test_suite: unit
# Job 1 (cont.): Rust tests, JUnit XML from cargo-nextest- name: Upload Rust results if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: target/nextest/ci/junit.xml commit_sha: ${{ github.sha }} branch: ${{ github.ref_name }} test_framework: cargo-nextest test_suite: rust-unit# Job 2: Playwright, native HTML report- name: Upload E2E results if: always() uses: gaffer-sh/gaffer-uploader@v2 with: gaffer_upload_token: ${{ secrets.GAFFER_PROJECT_TOKEN }} report_path: apps/dashboard/playwright-report commit_sha: ${{ github.sha }} branch: ${{ github.ref_name }} test_framework: playwright test_suite: e2eFormat detection is automatic, so there is no format argument to keep in sync. test_framework and test_suite are labels for filtering, not parsing hints.
If a framework has no natively parsed output, install a CTRF reporter and upload the JSON. CTRF reporters exist for Mocha, Jasmine, Go test, Cypress, WebdriverIO, .NET, Nightwatch, and others, and CTRF carries retries and per-test metadata that a JUnit XML conversion would drop.
Framework-specific setup
Each suite has its own reporter configuration and its own sharing quirks. The per-framework guides cover setup end to end:
- Playwright: native HTML report with traces, screenshots, and video
- Jest: built-in JSON,
jest-html-reporter, or the CTRF reporter - Vitest: JUnit, HTML, or CTRF reporters, plus Vitest coverage reports
- pytest:
pytest-html, JUnit XML, orpytest-ctrf - Cypress: CTRF via
cypress-ctrf-json-reporter, and the Cypress reports guide for Mochawesome and JUnit export - .NET: TRX from
dotnet test --logger:trx, covering MSTest, NUnit, and xUnit - Code coverage: LCOV, Cobertura, JaCoCo, and Clover alongside the test results
What you get
| Before | After |
|---|---|
| A pass rate per framework, none comparable | One pass rate per commit, computed on normalized statuses |
| Flaky detection only where the format supports it | Flaky detection on every suite, computed from run history |
| Retention that expires on a different clock per artifact | One retention window for every suite |
| A link per framework to share | One project URL, filterable by suite |
| ”Which layer is least reliable?” is unanswerable | A health score that moves when any suite moves |
Coverage rides along in the same view, so line coverage from LCOV and JaCoCo trends next to pass rate instead of in a separate tool.
Frequently asked questions
Can one dashboard show Playwright, Jest, pytest, and .NET results together?
Yes. Gaffer parses JUnit XML, Jest/Vitest JSON, native Playwright JSON, CTRF JSON, and TRX, and normalizes all of them to the same field set before storing them. Reports join on commit SHA, so every suite that ran against a commit appears under that commit regardless of which language or CI job produced it.
How is pass rate calculated when frameworks count skips and retries differently?
Gaffer maps every parsed test to one of five statuses: passed, failed, skipped, timedOut, or flaky. Formats that use several names for a skip are folded into one bucket, so Jest’s pending and todo and CTRF’s skipped, pending, and other all become skipped. Pass rate is computed on that normalized set rather than on each format’s own totals.
Do I have to convert everything to JUnit XML first?
No, and converting is usually a downgrade. JUnit XML has no retry count, no flaky flag, and no standard place for screenshots or browser metadata, so a Playwright suite converted to JUnit XML loses data Gaffer would otherwise keep. Upload each suite in its richest supported format and let the parser normalize.
Does Gaffer detect flaky tests in frameworks that have no flaky concept?
Yes. Flakiness is computed from history, not read from the report file. Gaffer groups executions by test name across runs, ignores anything that is not a pass or a fail, and needs at least five executions before it will judge a test. A test is flagged when it flips between pass and fail on at least 10% of consecutive run pairs, or when it fails with two or more distinct error messages after normalization.
What happens to a test that fails once and passes on retry?
It depends on the format, which is why the normalized view matters. Playwright JSON reports it in a separate flaky bucket, so it counts as neither passed nor failed in the run summary. Jest and Vitest JSON have no flaky bucket and report it as passed. Gaffer keeps each format’s own counts and separately recomputes flakiness across runs, so the two suites end up comparable.
Get started
Gaffer’s free tier includes 500 MB of storage with 7-day retention, which is enough to wire every suite in a polyglot repo into one project and see whether the numbers agree.