Engineering Practice
A Flaky Test Passed on Retry: Diagnose It Before You Trust the Build
A controlled Python example shows how shared test data can fail in one order and pass on a narrower retry, then checks a per-test fixture repair without claiming real CI stability.
When a failing test passes on retry, preserve both results and investigate what changed between the attempts before trusting the build. Compare the code, test order, worker process, data, timing, and environment. Use the retry to compare conditions and outcomes; it does not identify the cause by itself. If the failure exposes a real race or isolation defect, making the final build badge green can hide the problem rather than remove it.
This LandOffer.ai guide includes a completed failure-versus-rerun comparison and a small investigation we actually executed locally. The code is an original synthetic Python fixture, with no customer data, database, network, browser, or real CI service. Documentation was checked October 27, 2026. Results demonstrate one deliberately introduced test-state leak; they do not measure an organization's flake rate, validate a production application, or establish that any real build is safe to release.

Preserve the first failure before retrying
Save the failing test identifier, assertion or exception, complete relevant log, artifact revision, test command, runner and dependency versions, operating environment, seed where applicable, worker identity, and surrounding tests. Record whether the rerun reused the same worker process, started a new one, or selected a narrower group of tests. If it recreated a database, cleared a cache, changed timing, or selected only one test, that difference belongs in the comparison.
Do not overwrite the failed attempt with the successful retry. In a real system, retain authorized diagnostic evidence while redacting credentials, personal information, and unrelated application data. A short failure excerpt helps readers, but keep the full approved log available to the investigator. Without that evidence, another engineer may be unable to reconstruct the hidden dependency.
Google's May 2016 testing article defines flaky results as passing and failing with the same code and describes how false signals can cause genuine failures to be overlooked. It also discusses limitations of reruns and quarantine. Treat this as historical Google experience; its reported rates and retry policy are not current benchmarks for your team, and we do not use those figures here.
Record the retry policy as well as the individual outcomes. If a runner allows several attempts and reports success when one passes, its final result answers a different question from “did the first attempt pass?” Keep the number of attempts, which attempt failed, and whether the same assertion failed each time. A changed exception may indicate a different problem, even when the test name is unchanged. Do not combine skipped attempts, infrastructure errors, and assertion failures into one unexplained failure count.
First establish what “same code” covers. Application source, tests, fixtures, configuration, generated artifacts, and dependency locks may all matter. Unchanged application code does not prove unchanged conditions. A passing retry after the service recovered is also different from a deterministic local configuration mismatch; preserve the evidence before attaching a broad “flaky” label.
Turn possible causes into controlled comparisons
Investigate one plausible difference at a time. Does the test fail only after another test writes shared data? Does it depend on real time or completion of background work? Does parallel execution collide on a filename or account? Did an external service fail? These are hypotheses, not reasons to blame infrastructure by default.
The pytest documentation identifies uncontrolled system state, incomplete cleanup, ordering, and parallel execution as possible causes of unreliable results. It also discusses timing assertions and thread-related concerns. That is a useful investigation map, not evidence that one particular cause explains your failure. The demonstration below uses Python's standard-library unittest, not pytest or a pytest plugin.
Reduce the suite while keeping the failure-producing conditions. Keep the predecessor in the reproduction: a passing isolated test does not explain a failure caused by data left by another test. Record the preceding test and the shared resource rather than deleting assertions until the remaining command passes. Increasing a sleep can change how often a timing problem appears without proving the necessary condition is satisfied.
Choose diagnostics that fit the suspected mechanism. For a timing issue, inspect the awaited event and timeout evidence. For a state leak, inspect ownership and lifecycle of the shared fixture. For a random seed, preserve and replay it. A reproducible explanation for one mechanism should not become a claim that all nondeterminism has been removed.
Completed comparison: the same source behaves differently by order
We created two synthetic tests for an in-memory list representing draft records. The intended contract is that each test starts with its own empty list. The deliberately defective version instead uses a module-level list shared by both tests:
import unittest
RECORDS = []
class DraftTests(unittest.TestCase):
def test_add_draft(self):
RECORDS.append('synthetic-draft')
self.assertEqual(RECORDS, ['synthetic-draft'])
def test_starts_empty(self):
self.assertEqual(RECORDS, [])
We ran the fixture on Python 3.12.14, using a fresh subprocess for each command attempt. Both tests in a two-test command execute in that subprocess. The retained hash confirms the pre-fix source file is unchanged across all three comparisons; the controlled order and starting process differ. The retained investigation report includes source hashes, exact commands, runtime information, and the log for every attempt.
| Pre-fix command pattern | Deliberately controlled condition | Observed result across 10 command attempts |
|---|---|---|
| Add a draft, then check empty | Both tests share the module list in one fresh process | 0 suite passes; 10 suite failures, each at the empty-list assertion |
| Check empty, then add a draft | Reverse order in a fresh process | 10 suite passes; 0 failures |
| Check empty alone | No predecessor writes the list in the fresh process | 10 passes; 0 failures |
One failed log shows the empty-list assertion receiving ['synthetic-draft'] instead of []. The add test leaves that value in the module list. A fresh process running only the empty test starts with a newly initialized list, so that narrower rerun passes. The input state and executed sequence differ even though the source is unchanged.
These are controlled-order results, not random scheduling or real CI observations. We deliberately ran each pattern ten times. The counts do not estimate the probability of a failure in an uncontrolled suite. They also do not demonstrate a thread race: the example is sequential and its order-dependent failure is deterministic once the order is specified.
To reproduce the contrasting commands, save the defective snippet as before.py in an isolated directory with no production connection. From that directory, use:
python -m unittest -v before.DraftTests.test_add_draft before.DraftTests.test_starts_empty
python -m unittest -v before.DraftTests.test_starts_empty before.DraftTests.test_add_draft
python -m unittest -v before.DraftTests.test_starts_empty
Use the interpreter you intend to test and retain its version. The first command is expected to fail in this intentionally defective demonstration. Do not substitute your employer's suite, credentials, or shared test service into this exercise. The useful evidence is the contrast between the sequences, not the appearance of a green summary for the narrower retry.
Completed investigation: repair the fixture without weakening the assertion
The investigation record is now specific: the failure requires the add test to precede the empty test in the same process; both use the same module-level list; the failed assertion exposes the predecessor's synthetic value. Reversing order or starting a fresh process changes that shared state. This evidence supports a fixture-lifecycle defect in this small example, without establishing anything about an unrelated application failure.
We also check the boundary of the explanation. The fixture uses no threads, clock reads, external service, or random values. Those mechanisms cannot explain this deliberately small reproduction. They can still matter in a real failing suite, so this reduction does not rule them out elsewhere. The list value in the failure, combined with the order comparison, is the useful link between the hypothesis and the observation.
The chosen repair gives every test an empty instance-owned list through setUp(). Python's unittest documentation describes setup before each test and recommends self-contained test cases. We preserve the two expectations rather than accepting a dirty list or removing the failing check:
import unittest
class DraftTests(unittest.TestCase):
def setUp(self):
self.records = []
def test_add_draft(self):
self.records.append('synthetic-draft')
self.assertEqual(self.records, ['synthetic-draft'])
def test_starts_empty(self):
self.assertEqual(self.records, [])
Save that version separately as after.py. The same three command patterns can then use after.DraftTests instead of before.DraftTests. We actually executed each repaired pattern ten times in fresh subprocesses. All ten add-then-empty attempts passed, all ten reverse-order attempts passed, and all ten empty-alone attempts passed: 30 passing command attempts after the change.
The complete local evidence contains 60 command attempts across both versions. A two-test command runs two test methods; an empty-alone command runs one. Therefore these totals describe command attempts, not 60 individual test-method executions. The repaired source has its own hash and a retained diff; the original source and failure logs remain available for comparison.
The patch changes only this synthetic fixture's ownership. It does not add retries, loosen equality assertions, introduce arbitrary waits, or demonstrate a production-code fix. For real shared storage, isolation may require unique test data, correctly scoped transactions, controlled service state, and cleanup even after failure. An instance list is an adequate repair here because the entire example's state is that list, not because it solves database or distributed-system isolation generally.

Decide whether to fix, quarantine, or keep the gate blocked
After a passing retry, ask which behavior is still unverified. If the failure could represent data loss, an authorization error, a real race, or another material defect, the retry does not remove that concern. Follow the actual team's release and exception process. A local investigator should not silently grant permission to merge or deploy by changing a test marker.
Quarantine can reduce disruption, but it removes or changes a signal on the main path. If authorized, record the exact test and scope, reason, owner, repair issue, review date, and compensating coverage. Preserve its failures and continue running it where the team can see them. Do not label a skipped or nonblocking test as a verified pass, and do not leave a temporary exception without a return condition.
For this demonstration, the disposition is to retain the original failure evidence and use the isolated fixture version for the demonstrated checks. There is no real CI gate to quarantine or release owner to approve. The three exercised patterns support the specific state-isolation repair; they do not establish that every test order, parallel worker arrangement, Python version, or application behavior has been validated.
Repeated passes are evidence about the runs you performed, not proof that failure is impossible. Match follow-up work to the diagnosed mechanism and the real system's risks. If parallelism was implicated, investigate parallel execution; if timing was implicated, exercise the relevant timing conditions. More repeats of the same narrow scenario cannot substitute for missing coverage of a different cause.
Leave a diagnosis another engineer can verify
Write the handoff around the observation, controlled comparison, supported cause, repair, and remaining uncertainty. Include the commands and artifacts needed to reconstruct the reasoning. Say whether the original failure still reproduces, what changed in the repaired version, and whether the outcome concerns a synthetic local fixture, a real CI job, or a production defect.
The completed summary for this example is: “A module-level list made the empty-list test depend on its predecessor. The original add-then-empty sequence failed in all ten deliberately controlled attempts; the empty-only fresh-process rerun passed in all ten. Per-test instance setup removed that dependency in the three exercised patterns, each repeated ten times. No real application, parallel execution, or release readiness was assessed.”
Use that level of precision for your own investigation and keep the first failed result beside the retry. LandOffer.ai publishes this engineering guide; it does not inspect your CI system or certify a build. Give the decision owner the original failure, controlled reproduction, repair evidence, and remaining coverage gaps before changing a real build's status.


