Eval Run Setup is the moment before a test runs. The person picks the prompt versions, the models and the example set to test them against. The tool says what the run will cost and how long it will take before the person starts it. The run then carries a record of what was tested.
When to Use It
A run costs real money or real minutes
More than one version or model is in play
The team needs a record of what was tested
When Not To
Runs are free and instant; just run them
There is one version and one model; the choice is already made
The estimate would be a guess with no basis
Three views
The exchange between the human, the agent and the system the agent acts on; who holds each part of it; and the component that implements it.
01 · the interaction
02 · who holds what
01Prompt versionsThe versions to test, at least a baseline and a candidate
02ModelsThe models each version runs on
03Example setThe eval set the run uses, at a named version
04EstimateCost and time for the run, shown before the person commits
05Run recordWhat was tested, when, by whom, at what cost
03 · the component
eval run setup · primitives
Prompt versions
Modelsoncesessionalways
Example setEstimateRun record
primitive wireframe, generated from the anatomy — the installable component ships when this
pattern's anatomy stabilizes
A threshold is a decision made once, by whoever wrote the workflow file. The product does not ask whether this regression should block or only warn; that question needs a screen, not a setting.
A vendor retiring the surface moves the interaction into the team's own tools. That is an argument for patterns that outlive any one product, and for baselines the team owns.
Estimate honesty: a cost figure that is often wrong trains humans to ignore it. Show the range and the assumptions behind it.
Grid explosion: three versions times four models times two sets is a lot of runs. Defaults should make the smallest useful comparison the easy path.
Reruns: outputs vary run to run. One run of a small set can flip a verdict; the setup should say how many repeats it used.
The Story So Far
Sep 11, 2026Added to the Watching list from the prompt-evals brief and the 2026-09-11 vendor pass; the Chromatic review loop shapes the comparison and release entries.