1
# Why tests go flaky
2
3
## Order dependence and shared state
4
5
**Tell:** passes alone, fails in the suite, or fails only in a particular order.
6
7
Shared module state, a database not rolled back, a global registry, a cached singleton,
8
a file left behind, an environment variable set by another test.
9
10
**Fix:** make each test set up and tear down its own state. Randomize test order
11
permanently so this cannot silently return.
12
13
## Time
14
15
**Tell:** fails near midnight, at month boundaries, in a particular time zone, or on a
16
fast machine.
17
18
Tests that assert on `now`, assume a day is 24 hours, compare wall-clock durations, or
19
depend on daylight saving.
20
21
**Fix:** inject the clock. Never call the system clock from code under test. Use a fixed
22
instant in tests, and include an awkward one such as a leap day or a DST boundary.
23
24
## Concurrency
25
26
**Tell:** fails more under parallelism or on machines with more cores.
27
28
Real races: unsynchronized shared access, an assertion made before the work completes,
29
a callback that has not fired yet.
30
31
**Fix:** await the condition rather than a duration. If the race is in the product code
32
rather than the test, you have found a bug. Do not fix it in the test.
33
34
## Randomness
35
36
**Tell:** fails at a stable low rate with no environmental pattern.
37
38
Unseeded generators, random data that occasionally violates an assumption, hash
39
iteration order.
40
41
**Fix:** seed deterministically, and log the seed on every run. When a property-based
42
test finds a failing case, add that case as a permanent example test.
43
44
## Resource contention
45
46
**Tell:** fails only in CI, or only when several jobs run together.
47
48
Hardcoded ports, a shared temp path, a shared database, a fixed filename.
49
50
**Fix:** allocate dynamically. Port zero, a temp directory per test, a schema per worker.
51
52
## External dependencies
53
54
**Tell:** failures correlate with network conditions or another team's deploys.
55
56
A real HTTP call, a real DNS lookup, a shared staging environment, a rate limit.
57
58
**Fix:** stub at the boundary. Keep a small number of genuine integration tests, run
59
them separately, and never let them gate a unit test suite.
60
61
## A real bug
62
63
**Tell:** the failure is a legitimate assertion failure that only occurs under a
64
specific interleaving or input.
65
66
**Fix:** fix the product. This is the case where the flaky test was doing its job, and
67
the retry annotation would have hidden a defect that will eventually reach production.
68