Case Drilldown is one example examined on its own. The person reads both answers and the reasoning behind the verdict of the judge, the model that scored them. They can overrule the verdict with a note, and they can keep the case as a permanent test so the failure is caught next time. Overrules are counted, because they show where the judge and humans disagree.
When to Use It
A verdict looks wrong and someone needs to say so
A failure should never happen again
The team wants to know how much to trust the judge
When Not To
Cases are read-only; then it is a report, not a drilldown
There is no judge reasoning to show
Overrules go nowhere; then collect them elsewhere
Three views
The exchange between the human, the agent and the system the agent acts on; who holds each part of it; and the component that implements it.
01 · the interaction
02 · who holds what
01The caseOne input and the answers it produced
02Judge verdictPass or fail, with the reasoning in words
03OverrideThe person sets their own label, with a note
04PromoteThe case joins the permanent test set
05Disagreement countHow often humans overrule the judge, kept in view
03 · the component
case drilldown · primitives
The case
Judge verdict
Override
Promote
Disagreement count
primitive wireframe, generated from the anatomy — the installable component ships when this
pattern's anatomy stabilizes
A score you cannot open is a score you cannot trust. The judge's reasoning, and the cases where humans overruled it, belong on the screen beside the number.
This is the number behind "calibrate with a few labels". A product can tell the reviewer how many cases to label before the judge can be trusted. Not all of them, and not none.
The alignment score is the trust signal. A judge version with a number beside it, measured against the team's own grades, is something a product manager can decide to rely on.
Ten at a time is the right unit. It is enough to calibrate a judge and small enough that a designer will do it over coffee. That is the difference between a habit and a study.
Sep 11, 2026Added to the Watching list from the prompt-evals brief and the 2026-09-11 vendor pass; the Chromatic review loop shapes the comparison and release entries.