1
# Incident Runbook — Workflow
2
3
A structured procedure for a production incident: triage, mitigate, communicate, then
4
postmortem. Timers and escalation rules keep the response moving instead of stalling
5
on investigation while impact continues.
6
7
Inputs: `severity` (required — `sev1`|`sev2`|`sev3`), `incident_channel` (optional),
8
`escalation_contact` (optional).
9
10
## Severity definitions and timers
11
12
| Severity | Definition | First status update | Escalate if not mitigated within |
13
|---|---|---|---|
14
| **sev1** | Full outage or critical data-integrity risk; all/most users affected | within 5 min | 30 min |
15
| **sev2** | Major functionality degraded or a significant subset of users affected | within 15 min | 2 hours |
16
| **sev3** | Minor/limited impact, workaround available | within 60 min | next business day |
17
18
Re-classify severity if new information changes the picture (escalate up if impact
19
turns out wider than first thought; de-escalate down and say so explicitly if it
20
turns out narrower — don't just quietly stop treating it as urgent).
21
22
## Phase 1 — Triage
23
24
1. **Confirm it's real.** Check monitoring/alerts against actual user-facing behavior
25
   before mobilizing a full response — false alarms happen (a broken alert threshold,
26
   a synthetic check hitting a genuinely-down staging endpoint).
27
2. **Establish impact.** What's broken, for whom, since when. Prefer concrete signals
28
   (error rate graph, affected request count) over guesses.
29
3. **Post the first status update** within the severity's window, using
30
   `templates/status-update.md`, to `incident_channel` if given. Silence during an
31
   active incident is worse than an update that just says "still investigating."
32
4. **Start the incident timeline.** A running, timestamped log of what was observed,
33
   tried, and found — this becomes the postmortem's factual backbone, so keep it
34
   contemporaneous rather than reconstructing it afterward.
35
5. **Form a hypothesis** from recent changes: deploys, config/flag changes, dependency
36
   upgrades, infra changes, or traffic pattern shifts in the relevant window. Check
37
   `git log`/deploy history for anything that shipped shortly before impact started —
38
   correlation here is a strong starting lead, not proof.
39
40
## Phase 2 — Mitigate
41
42
Priority order: **stop the bleeding before finding root cause.** A fast rollback that
43
resolves user impact is a better first move than a slow, thorough root-cause
44
investigation while users are still affected.
45
46
1. If a recent deploy/change correlates with the incident start, **roll it back**
47
   first, and confirm impact actually stops before declaring the rollback the fix
48
   (correlation ≠ causation — verify).
49
2. If rollback isn't applicable (e.g. infra failure, third-party outage), consider:
50
   failover to a backup region/provider, feature-flagging off the affected path,
51
   scaling up a starved resource, or restarting an unhealthy process — whichever
52
   restores service fastest with the least additional risk.
53
3. **Every mitigation action is logged in the timeline** with timestamp and who/what
54
   performed it, before moving to the next action — this matters both for the
55
   postmortem and in case the action itself needs to be undone.
56
4. If the severity's escalation window (see table) passes without mitigation,
57
   escalate to `escalation_contact` now — don't wait for a "good stopping point."
58
   Escalating and then resolving it yourself a minute later costs little; not
59
   escalating on a stuck sev1 costs a lot.
60
5. Once impact is confirmed resolved (not just "the fix is deployed" — check the
61
   actual signal that indicated impact), post a mitigation status update and move to
62
   Phase 3.
63
64
## Phase 3 — Communicate
65
66
- Status updates go out at the cadence implied by severity: sev1 every 30 min until
67
  resolved, sev2 hourly, sev3 as material updates occur.
68
- Use `templates/status-update.md` for consistency: what's known, what's affected,
69
  what's being done, next update time.
70
- State facts and current status; avoid speculating on root cause publicly until it's
71
  confirmed — a wrong public guess has to be walked back and erodes trust more than a
72
  plain "investigating" would have.
73
- Post a final resolved update explicitly — don't let updates just trail off.
74
75
## Phase 4 — Postmortem
76
77
Within a business day or two of resolution (while details are fresh), write the
78
postmortem using `templates/postmortem.md`. Blameless: focus on what in the system and
79
process allowed the incident, not who made a mistake — a person following a
80
reasonable process that still led to an incident is a process/systems finding, not a
81
personal one.
82
83
Every postmortem produces concrete, owned, tracked action items — a postmortem with
84
only narrative and no action items is incomplete.
85
86
## Stop conditions / escalation triggers
87
88
- Escalation window from the severity table passes without mitigation → escalate
89
  immediately per Phase 2.
90
- Mitigation attempt makes things worse (verified by the impact signal, not
91
  assumption) → roll back the mitigation itself and escalate; don't keep trying
92
  variations of a failing approach without a fresh pair of eyes.
93
- Impact is confirmed fully resolved → move to Phase 3's final update and schedule the
94
  Phase 4 postmortem; the incident is not "done" until the postmortem's action items
95
  are filed (even if not yet completed).
96