Eval Set Authoring is how a person builds the examples a prompt is tested against. The examples come from real traffic, from generated cases, or from paste. Beside them the person writes the rubric: plain sentences that say what a good answer does and does not do. The rubric is a shared object, edited and versioned with the prompt.
When to Use It
A prompt drives behavior real humans see
Quality lives in someone's head and needs to be written down
You have real inputs to draw examples from
When Not To
The prompt is a one-off; a handful of manual checks is enough
Nobody can say what good looks like yet; run a few cases by hand first
The examples would expose private data with no plan to handle it
Three views
The exchange between the human, the agent and the system the agent acts on; who holds each part of it; and the component that implements it.
01 · the interaction
02 · who holds what
01SourceWhere the examples come from: real traffic, generated cases, or paste
02CasesThe examples themselves, each with an input and, where known, a good answer
03RubricPlain sentences that say what a good answer does and does not do
04Pass and fail examplesOne answer that passes and one that fails, beside each rule
05VersionThe set and the rubric carry a version, beside the prompt version
03 · the component
eval set authoring · primitives
Source
Cases
Rubric
Pass and fail examples
Version
primitive wireframe, generated from the anatomy — the installable component ships when this
pattern's anatomy stabilizes
This is the number behind "calibrate with a few labels". A product can tell the reviewer how many cases to label before the judge can be trusted. Not all of them, and not none.
Writing the test set was the chore that kept teams iterating by feel. Describing the failure in a sentence and getting rows and a scorer back is the step that turns the chore into a habit.
The alignment score is the trust signal. A judge version with a number beside it, measured against the team's own grades, is something a product manager can decide to rely on.
A rubric per example hides a cost: a standard that shifts with each case is hard for a team to share or defend. The pass and fail examples beside each rule are what let a team hold the rubric steady across cases.
Similarity = Jaccard overlap of evidenced anatomy sets, Anthropic vs OpenAI, cumulative by
month · n = 2 examples. Dashed spans: n < 3 — not enough evidence yet.
View as table
Month
Shared anatomy %
Evidence rows
2024-07
0
1
2025-10
100
1
Tensions & Failure Modes
Rot: a test set written for last quarter's product tests last quarter's behavior. Sets need a way to age out cases and add new ones from production.
One person's taste: a rubric only its author can apply is a preference, not a standard. Pass and fail examples beside each rule make it shared.
Synthetic drift: generated cases are cheap and plentiful, and they can miss how real humans phrase things. Real traffic keeps the set honest.
The Story So Far
Sep 11, 2026Added to the Watching list from the prompt-evals brief and the 2026-09-11 vendor pass; the Chromatic review loop shapes the comparison and release entries.