Back to Devoured - November 20, 2026
DEVOURED
The 2026 Flaky-Test Tooling Landscape: An Honest Comparison

The 2026 Flaky-Test Tooling Landscape: An Honest Comparison

DevOps Harness
An IEEE study confirms that up to 16.3% of failed CI pipelines contain missed flaky failures, rendering simple rerun-based detection unreliable.
What: Harness evaluates seven flaky-test tools, concluding that detection accuracy is secondary to effective quarantine and recovery workflows, with most tools offering varying interpretations of 'quarantine'.
Why it matters: Engineering teams often stop trusting CI results when failures are not clearly distinguished as code defects versus infrastructure/test instability, leading to 'broken window' syndrome in pipelines.
Takeaway: Evaluate your flaky-test tool by how it handles reinstatement; prioritize tools that quarantine tests without silencing the signal, or those that automate recovery based on test-health metrics.
Deep dive
  • Research: 2026 IEEE study on 8.8 billion executions indicates rerun-based detection misses nearly 1/6th of actual flaky failures.
  • The Quarantine Problem: 'Quarantine' inconsistently means either 'run and report failures without blocking' or 'skip execution entirely', with the latter hiding potential recovery.
  • Harness Approach: Uses a 14-day same-commit window for detection and requires 5 consecutive passes for automatic reinstatement.
  • Buildkite: Prefers a 'mute' vs 'skip' model to manage CI budgets vs reliability.
  • CircleCI: Recently GA'd automatic reruns and test atom pricing.
  • Evaluation Criteria: Focus on identifying how a tool defines 'flakiness', what it does during quarantine, how it handles ownership, and its true total cost of ownership.
Decoder
  • Flaky Test: A test that produces inconsistent results (passing or failing) for the same code state, typically due to environmental non-determinism.
  • Test Shard: A logical subset of a test suite executed in parallel to reduce total pipeline duration.
  • Quarantine: A policy that flags unstable tests to prevent them from blocking deployment while maintaining observation.
Original article

The 2026 Flaky-Test Tooling Landscape: What Independent Reviews Actually Say

Seven flaky-test tools compared in 2026, grounded in independent research. What Harness, Buildkite, CircleCI, Trunk and others really do.

TL;DR

  • Test time is not pipeline time. Measure where the time actually goes before changing the test runner or the retry policy.
  • A green rerun is not a fix. Quarantine is the honest middle path when the test keeps running and the signal survives.
  • Fix state and ingestion first. Only after tests are deterministic do parallel shards, selective execution, and the Harness agent's test-health policies pay off.

A test fails in CI. You rerun the pipeline without touching a single line of code, and this time everything passes. The commit ships. Nobody files a ticket.

Three weeks later, the same test fails on a different branch. Then on main. By the time anyone looks, it has been flaking quietly for a month, and nobody can agree whether it ever caught a real regression or just burned CI minutes.

This is what makes flaky tests expensive. The direct cost of rerunning them is real, but the compounding cost is reader trust. Once a team stops believing a red pipeline, they stop blocking merges on it, and the whole point of CI collapses from the inside.

By 2026, the market has responded. CI platforms ship same-commit flaky detection and quarantine policies. Dedicated reliability vendors sell detection and recovery workflows as a layer on top of your existing CI. Test-observability platforms wire the whole lifecycle into their broader instrumentation. Every vendor page promises better reliability. The useful question is harder: what does independent evidence actually say about any of this?

Independent research says reruns are not enough

The dominant detection approach in the literature and in most vendor products is still dynamic rerun-based: run a test enough times and inconsistent outcomes expose flakiness. Recent independent studies suggest that is a shakier foundation than product pages imply.

A 2026 study in IEEE Transactions on Software Engineering analyzed 8.8 billion test executions across four industrial projects and found that 9.8% to 16.3% of failed pipeline runs involved flaky failures that existing detection mechanisms missed. Failure rates also varied by as much as three times between environments, meaning the test environment itself is a variable the detection logic is usually blind to.

A 2026 evaluation of Shaker for Python projects pushed on this further. Across 137 paired tests, Shaker's stress-based detection caught 37.2% of known flaky cases, ordinary reruns caught 35.8%, and the difference was not statistically significant.

Detection methodology matters less than knowing what your tool misses. The real differences across vendors are not whether they detect flaky tests; they are what happens next: how a flaky test is quarantined, what evidence survives quarantine, how a team learns the test has been fixed, and what adoption actually costs.

The seven tools in play

The 2026 landscape sorts into three groups. CI-native capabilities inside the platform you already use (Harness CI, Buildkite Test Engine, CircleCI). A dedicated reliability layer on top of any CI (Trunk, BuildPulse, Mergify). A test-observability model inside a broader platform (Datadog).

Tool Detection What quarantine does Primary focus
Harness CI Same-commit inconsistency Test keeps running, failures excluded from pipeline status CI-native reliability
Buildkite Test Engine Historical monitors Muted (runs, failures ignored) or Skipped (does not run) Configurable test state
CircleCI Same-commit plus auto reruns Failed-test retry, test-atom metered CI-integrated optimisation
Trunk Flaky Tests Flakiness monitors Keeps running without blocking the job Dedicated reliability layer
BuildPulse Git-based plus statistical Quarantine list exposed via API, CI skips Impact and prioritization
Mergify Test Engine Default-branch plus PR reruns Quarantine with continued reporting Prevention before merge
Datadog Test Optimization Historical plus Early Flake Detection Active, Quarantined, Disabled, Fixed lifecycle Reliability inside observability

The Harness agent: flaky management inside the pipeline

The Harness agent treats flaky-test management as part of normal CI, not a separate product. That matters because every other tool on this list is either an add-on with its own ingestion contract or a workflow that only lights up above a plan tier. The Harness agent ships detection, test-health visibility, and quarantine inside the stage the test already runs in.

Per the Flaky Test Detection docs, a test is flaky when it has both passed and failed on the same commit inside a 14-day observation window. It clears after five consecutive successful executions or when no inconsistency has occurred for 14 days. The 14-day window is the same window CircleCI uses, which is useful because detection semantics are at least consistent across the CI-native options.

Test Management layers test-health visibility and quarantine on top. A quarantined test keeps running, its results still land in the dashboard, and its failures can be configured to stop breaking the pipeline.

Buildkite Test Engine: Muted vs Skipped is the whole decision

Buildkite builds flaky management around test state. The test-state docs define two behaviors. Muted tests keep running, their failures do not block the build, and data keeps flowing. Skipped tests do not run at all, which means the system cannot tell you whether they have recovered. Buildkite's own guidance is to prefer muting over skipping unless a test must not run. Skipping is a budget decision, not a reliability one.

CircleCI: automatic reruns went GA this October

CircleCI's Test Insights docs use the same same-commit, 14-day definition of flakiness Harness does. The 2026 change is Smarter Testing. On 13 October 2026 CircleCI announced general availability of a bundle including automatic reruns, intelligent test selection, dynamic splitting, and analytics.

Automatic reruns answer one question: is this failure reproducible? They are not a quarantine lifecycle. A test that passes on rerun is marked a potential flake, and the build continues. There is no notion of ownership, no automated reinstatement after N clean runs.

Trunk Flaky Tests: separating flaky from broken

Trunk builds flaky-test reliability as a product in its own right. Through Trunk's quarantine, known flaky tests keep running and reporting without blocking the job. The important design choice is one Trunk is explicit about: automated quarantine applies to tests already identified as flaky, not to tests that are consistently broken, because quarantining them hides a real regression.

BuildPulse: measuring the cost of flakiness

BuildPulse puts the operational cost of flaky tests front and centre. Its Flaky Tests Dashboard documents Git-based detection, which compares inconsistent outcomes against equivalent Git tree states, and statistical detection against thresholds. Beyond detection, BuildPulse surfaces metrics most CI platforms do not: disruption frequency per test, failed builds attributable to specific flakes, and engineering time lost.

Mergify Test Engine: catch flakes before they merge

Mergify splits the problem into Prevention, Detection, and Quarantine. Detection looks at test results on the default branch to find existing unreliable tests. Prevention operates on pull requests: with the right framework integration and a configured rerun budget, it reruns new or modified tests on the PR to expose inconsistent behaviour before merge.

Datadog Test Optimization: observability for tests

Datadog's Flaky Tests Management docs model a test's state as Active, Quarantined, Disabled, or Fixed, and track transitions. A quarantined test keeps its history, so when the team eventually investigates, the full timeline is there. Early Flake Detection proactively reruns previously unseen tests, similar in spirit to Mergify Prevention, wired into the same instrumentation that already watches your services.

The four questions that beat a feature matrix

How does the tool identify a flaky test? Same-commit inconsistency, PR reruns, and statistical detection against thresholds are three different signals. Each catches a different subset of flakes and misses a different subset.

What exactly does quarantine do? Quarantine is one word doing two very different jobs. In one version, the test keeps running, its failures stop blocking merges, and the signal survives. In the other, the test does not run at all, which saves time but destroys the only mechanism that would tell you it recovered. The second model is cheaper and more dangerous.

How does the team learn a test is fixed? Detection is where every vendor markets. Recovery is where most of them go quiet. Who owns the quarantined test, how is ownership tracked, what gets reinstated automatically, what demands manual review.

What does adoption actually require? A vendor supporting JUnit reports does not necessarily support quarantine on JUnit reports. Advanced detection often needs a native agent, a minimum library version, or an account feature flag.

Which approach fits which team

There is no independently established universal winner across these seven products.

Teams that want detection, test-health visibility, and quarantine inside the CI stage, without maintaining a second ingestion contract on top, get the most compact answer from the Harness agent.

Teams on another CI who do not want to move it, but do want governance on top, split between Trunk (quarantine with a clear flaky/broken distinction), BuildPulse (impact measurement and API-driven skipping) and Mergify (prevention before merge).

Teams already deep in Datadog for application observability get the test lifecycle in the same instrumentation, with native-agent coverage as the trade-off to verify.

A trustworthy pipeline is more than a green pipeline

Flaky-test management in 2026 has moved past "run it twice." Tools now stitch together detection, proactive reruns, quarantine, test-health dashboards, ownership, and recovery. The research is clear on one thing: flakiness is hard to detect completely, and every detection technique has conditions under which it falls over. The useful tools are the ones that do more than hide failed builds. They help developers understand what went wrong, keep an audit trail on unreliable tests, and give the team a defensible way to restore confidence once a test is actually fixed.

The objective is not to make every pipeline green. It is to make a green pipeline mean something.

Explore all 56 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi