1
# Performance Investigation
2
3
## 1. Define slow, in numbers
4
5
"The page is slow" cannot be finished. Get to a statement that can be:
6
7
- Which operation, with which inputs?
8
- What is it now, at p50, p95 and p99? An average alone hides the problem you were
9
  called about.
10
- What would be acceptable? Without a target you cannot know when to stop.
11
- Is it slow always, or under a condition? Load, data size, a particular tenant, a cold
12
  cache. The condition is frequently the whole answer.
13
14
## 2. Baseline before touching anything
15
16
Record a reproducible measurement. See `measurement.md`. Without a baseline you cannot
17
tell an improvement from noise, and every later claim is unfalsifiable.
18
19
Capture the environment too: machine, data size, concurrency, cache state, versions. A
20
baseline that cannot be reproduced is not a baseline.
21
22
## 3. Profile. Do not theorize
23
24
Get a profile before forming a hypothesis. Intuitions about where time goes are wrong
25
often enough that the profile is always cheaper than the argument.
26
27
Match the tool to the shape:
28
29
- **CPU-bound:** a sampling profiler, read as a flame graph.
30
- **I/O-bound:** trace the calls. Count queries, requests, file reads.
31
- **Distributed:** a trace across services, to find which hop owns the time.
32
- **Memory-driven:** allocation profile, plus garbage collection pauses.
33
34
Read the profile for **total inclusive time**, not for what looks inefficient. A tidy
35
function called two million times beats an ugly one called twice.
36
37
## 4. Find the actual top cost
38
39
Usually one of these, and they are worth checking in order:
40
41
1. **Repeated work.** The N+1 query, the same request in a loop, a computation not
42
   memoized. The most common cause by a wide margin.
43
2. **Work not needed.** Fetching columns nobody reads, serializing a field nobody uses,
44
   sorting a list that gets filtered afterwards.
45
3. **Wrong data structure.** A linear scan where a hash lookup belongs, quadratic
46
   behavior that was fine at ten items and is not at ten thousand.
47
4. **Waiting.** Sequential calls that could overlap. Look for a series of awaits with no
48
   dependency between them.
49
5. **Serialization boundaries.** JSON encode and decode, ORM hydration, network hops.
50
51
## 5. Change one thing
52
53
One change, then re-measure. Batched changes make it impossible to attribute the
54
result, and one of them is usually making things worse while another hides it.
55
56
Prefer, in order: delete the work, do it once instead of N times, do it concurrently,
57
do it faster. Deleting work always wins, and it is the option people consider last.
58
59
## 6. Verify honestly
60
61
Re-run the same measurement, same environment, enough repetitions to see past noise.
62
63
Report:
64
65
- Before and after at p50, p95 and p99, with the same units.
66
- Whether the difference exceeds run-to-run variance. If it does not, it is not a result.
67
- **What got worse.** Nearly every optimization trades something: memory, cold-start,
68
  code clarity, a different workload. Name the trade.
69
- Whether the goal from step 1 is met, or how far short it falls.
70
71
If the change did not help, revert it. An optimization that does not measurably help is
72
pure cost, paid by every future reader.
73
74
## 7. Stop at the goal
75
76
Stop when the target is met. Continuing past it trades readability for numbers nobody
77
needed, and the next person to read the code pays for it.
78