1
# Choosing and trusting a grader
2
3
Match the grader to the claim. Reach for the cheapest one that can actually decide.
4
5
| Case asks | Grader | Notes |
6
|---|---|---|
7
| Exact value | string or JSON equality | Normalize whitespace and key order first. |
8
| Structure | schema validation | Validate, do not regex. Report which field failed. |
9
| Code works | run it | Compile, run the test, check the exit code. Strongest grader there is. |
10
| A property holds | assertion in code | Cited path exists, no secret in output, under N tool calls. |
11
| Judgment | LLM grader | Weakest. Use only when the three above genuinely cannot decide. |
12
13
## Keeping an LLM grader honest
14
15
An LLM grader is a model with an opinion, and it will happily agree with whatever it
16
is shown. Constrain it:
17
18
1. **Give it the rubric, not the goal.** "Does the answer cite a file that exists?"
19
   beats "is this a good answer?"
20
2. **Make it output a verdict token plus a reason**, one of `pass`, `fail`, `unknown`,
21
   and one sentence. Parse the token. Read the reasons when triaging.
22
3. **Never show it which output came from the new version.** Order-swap A/B pairs.
23
4. **Calibrate it against humans.** Hand-label 20 cases yourself. If the grader
24
   disagrees with you on more than 2, fix the grader before trusting a single score.
25
5. **Re-calibrate when you change the grader model.** It is a dependency like any other.
26
27
## The unknown bucket
28
29
Any grader may return `unknown`: the output was malformed, a tool errored, the case
30
was ambiguous. Treat it as a defect in the harness, not a neutral result.
31
32
- `unknown` never counts as a pass.
33
- A suite over about 10% unknown is not measuring anything yet. Fix that before
34
  reading scores.
35