1
# Flaky Test Triage
2
3
## 1. Establish the rate before touching anything
4
5
"Sometimes fails" is not a measurement. Run it `runs` times (default 50) and record the
6
failure rate.
7
8
```bash
9
# Adapt to your runner. The point is repetition and a count.
10
for i in $(seq 1 50); do <run one test> >/dev/null 2>&1 || echo "fail $i"; done | wc -l
11
```
12
13
Run it three ways, because the differences are the diagnosis:
14
15
| Condition | What a failure here implies |
16
|---|---|
17
| Alone, repeated | Self-contained nondeterminism: time, randomness, ordering |
18
| With the full suite | Shared state or pollution from another test |
19
| Under parallelism | Resource contention: ports, files, database rows |
20
21
A test that only fails in one of the three has already told you which family of cause
22
you are in.
23
24
## 2. Capture a failure with enough detail to read
25
26
Do not debug from the assertion message alone. On failure, capture:
27
28
- The full diff between expected and actual, not a truncated boolean.
29
- Timestamps at each step, to spot a timing edge.
30
- The random seed and the test execution order.
31
- Relevant environment: time zone, locale, machine, concurrency level.
32
33
If your runner cannot report the seed and the order, fix that first. Without it, an
34
ordering bug is unreproducible by construction.
35
36
## 3. Classify the cause
37
38
Work through `causes.md`. Match on the tell, not on intuition. The families, roughly in
39
order of frequency:
40
41
1. Order dependence and shared state
42
2. Time and clock assumptions
43
3. Concurrency and race conditions
44
4. Unseeded randomness
45
5. Resource contention
46
6. External dependencies
47
7. Genuine product bugs that only surface under a specific interleaving
48
49
The last one matters most: **a flaky test is sometimes a correct test finding a real
50
race.** Rule that out before assuming the test is at fault. Silencing it would be
51
deleting a bug report.
52
53
## 4. Prove the cause
54
55
A hypothesis is not confirmed until you can turn the failure on and off:
56
57
- Make it fail every time. Force the ordering, pin the clock, hold the lock, use the
58
  seed that fails. If you cannot make it fail deterministically, you have not found it.
59
- Then apply the fix and make it pass every time.
60
61
Skipping this step is how a test gets "fixed" three times and stays flaky.
62
63
## 5. Fix the cause, not the symptom
64
65
Ranked, best first:
66
67
1. **Remove the nondeterminism.** Inject the clock, seed the randomness, fix the shared
68
   state, await the actual condition.
69
2. **Make the test wait for a condition, never a duration.** `sleep` is a race with
70
   extra steps.
71
3. **Isolate the resource.** A unique port, a temp directory, a per-test schema.
72
4. **Narrow the assertion** if it was over-specifying, for example asserting an
73
   unordered collection in order.
74
75
Never acceptable as a fix: a retry annotation, a longer sleep, or a loosened assertion
76
that would no longer catch the original bug.
77
78
## 6. Quarantine honestly, if you must
79
80
Sometimes the fix cannot land today. Then:
81
82
- Move the test out of the blocking suite, but keep running it and keep reporting it.
83
- File an issue with the failure rate you measured and everything you established.
84
- Put an expiry on it. A quarantine with no date is a deletion with extra steps.
85
- Never quarantine a test whose failure you have not yet explained. That is where a
86
  real bug goes to be forgotten.
87
88
## 7. Verify
89
90
Re-run the same three conditions from step 1, at the same repetition count. Report the
91
before and after rates as numbers. "Seems better" is not a result.
92