Performance Investigation

1. Define slow, in numbers

"The page is slow" cannot be finished. Get to a statement that can be:

• Which operation, with which inputs?
• What is it now, at p50, p95 and p99? An average alone hides the problem you were called about.
• What would be acceptable? Without a target you cannot know when to stop.
• Is it slow always, or under a condition? Load, data size, a particular tenant, a cold cache. The condition is frequently the whole answer.

2. Baseline before touching anything

Record a reproducible measurement. See measurement.md. Without a baseline you cannot tell an improvement from noise, and every later claim is unfalsifiable.

Capture the environment too: machine, data size, concurrency, cache state, versions. A baseline that cannot be reproduced is not a baseline.

3. Profile. Do not theorize

Get a profile before forming a hypothesis. Intuitions about where time goes are wrong often enough that the profile is always cheaper than the argument.

Match the tool to the shape:

• CPU-bound: a sampling profiler, read as a flame graph.
• I/O-bound: trace the calls. Count queries, requests, file reads.
• Distributed: a trace across services, to find which hop owns the time.
• Memory-driven: allocation profile, plus garbage collection pauses.

Read the profile for total inclusive time, not for what looks inefficient. A tidy function called two million times beats an ugly one called twice.

4. Find the actual top cost

Usually one of these, and they are worth checking in order:

1. Repeated work. The N+1 query, the same request in a loop, a computation not memoized. The most common cause by a wide margin.
2. Work not needed. Fetching columns nobody reads, serializing a field nobody uses, sorting a list that gets filtered afterwards.
3. Wrong data structure. A linear scan where a hash lookup belongs, quadratic behavior that was fine at ten items and is not at ten thousand.
4. Waiting. Sequential calls that could overlap. Look for a series of awaits with no dependency between them.
5. Serialization boundaries. JSON encode and decode, ORM hydration, network hops.

5. Change one thing

One change, then re-measure. Batched changes make it impossible to attribute the result, and one of them is usually making things worse while another hides it.

Prefer, in order: delete the work, do it once instead of N times, do it concurrently, do it faster. Deleting work always wins, and it is the option people consider last.

6. Verify honestly

Re-run the same measurement, same environment, enough repetitions to see past noise.

Report:

• Before and after at p50, p95 and p99, with the same units.
• Whether the difference exceeds run-to-run variance. If it does not, it is not a result.
• What got worse. Nearly every optimization trades something: memory, cold-start, code clarity, a different workload. Name the trade.
• Whether the goal from step 1 is met, or how far short it falls.

If the change did not help, revert it. An optimization that does not measurably help is pure cost, paid by every future reader.

7. Stop at the goal

Stop when the target is met. Continuing past it trades readability for numbers nobody needed, and the next person to read the code pays for it.