1
# Agent Eval Harness
2
3
Builds an eval suite that can actually catch a regression. The output is a case file,
4
a grader per case type, a recorded baseline, and a gate that fails a change which
5
makes things worse.
6
7
## 0. Decide what better means, before looking at any output
8
9
Write down the one or two things this agent must get right, in a sentence each. If you
10
cannot state them, you cannot grade them, and the eval will drift into measuring
11
whatever is easy to measure.
12
13
Bad: "responses are high quality."
14
15
Good: "the cited file path exists in the repo" and "the fix compiles."
16
17
## 1. Collect cases from reality, not imagination
18
19
Aim for `n_cases` (default 30). A suite of 30 real cases beats 300 synthetic ones.
20
In priority order:
21
22
1. **Bugs.** Every reported failure becomes a case, with the expected behavior as the
23
   report described it. These are the highest-value cases you will ever have.
24
2. **Production traffic.** Sample real inputs. Stratify. Take some from the head and
25
   some from the tail, not 30 of the same shape.
26
3. **Known-hard cases.** Inputs already near the edge of capability.
27
4. **Synthetic cases, last.** Only to cover a branch the first three missed.
28
29
Record each case in the `templates/cases.jsonl` format. Every case needs an `id`, an
30
`input`, and enough of an `expected` to grade against.
31
32
## 2. Split the suite
33
34
- **dev** (about 60%): you look at these, iterate against them, and overfit them. Fine.
35
- **held-out** (about 40%): you do not look at individual outputs, only the aggregate.
36
37
Report both numbers, always. A dev score that climbs while held-out stays flat means
38
you tuned to the cases, not the task.
39
40
## 3. Write a grader per case type
41
42
See `grading.md`. The rule that matters: a grader that cannot judge a case must return
43
`unknown`, never `pass`. Silent passes are how eval suites rot into decoration. Track
44
`unknown` as its own bucket and drive it toward zero.
45
46
## 4. Take a baseline before changing anything
47
48
Run the current version against the full suite and record:
49
50
- pass, fail and unknown counts, for dev and held-out separately
51
- the score per case type, not just the total
52
- cost and p50/p95 latency per case
53
- the run date, model id, and prompt version
54
55
Commit this. A baseline you cannot reproduce is a rumor.
56
57
## 5. Gate changes on it
58
59
A change ships when held-out pass rate does not drop and no case type regresses. Two
60
rules make this survive contact with reality:
61
62
- **Investigate every newly failing case individually.** A steady aggregate can hide
63
  two cases breaking while two unrelated ones start passing.
64
- **A flaky case is a bug in the case or the grader.** Run the suite twice against an
65
  unchanged target. Anything that flips is not measuring the target. Fix it or drop it.
66
67
## 6. Keep it alive
68
69
- Add the case first when a bug is reported, and watch it fail. Then fix.
70
- Re-baseline on a model or major prompt change, and note why in the commit.
71
- Prune cases that have never once failed and never will.
72
73
## Reporting
74
75
State the score honestly:
76
77
> held-out 24/30 pass, 4 fail, 2 unknown (baseline: 22/30, 6 fail, 2 unknown).
78
> Regression on tool-choice cases: 3/5 down to 2/5.
79
80
Never report a single percentage with no denominator, no held-out split, and no
81
unknown count. That number is unfalsifiable and therefore useless.
82