Production Drift measures live answers against the same rubric the tests used. The person sees quality over time, not only at release. A slip shows up as a change against the standard, with the cases behind it. Failures found here become new test cases.
When to Use It
The prompt runs in production at volume
Inputs change faster than the eval set
A slip would be felt by real humans before anyone looked
When Not To
Volume is too low to see a trend
Scoring would send private data to a model with no plan for it
No rubric exists; production scoring without a standard is a dashboard
Three views
The exchange between the human, the agent and the system the agent acts on; who holds each part of it; and the component that implements it.
01 · the interaction
02 · who holds what
01SampleWhich live answers are scored, and how many
02Same rubricThe standard the tests used, applied to live traffic
03TrendQuality over time, by dimension
04AlertA slip past a threshold reaches the person
05Back to the setA production failure becomes a test case
03 · the component
production drift · primitives
Sample
Same rubric
Trend
Alert
Back to the set
primitive wireframe, generated from the anatomy — the installable component ships when this
pattern's anatomy stabilizes
The brief you write for an agent shapes its behavior only where that behavior can be checked. Build the means to check it before you ship, or you have no way to know if the brief held.
Three words on the release check beat a dashboard that goes unopened. The open question is who decides: failing the job on a regression is a setting, and a person still owns the call.
Writing the test set was the chore that kept teams iterating by feel. Describing the failure in a sentence and getting rows and a scorer back is the step that turns the chore into a habit.
Ten at a time is the right unit. It is enough to calibrate a judge and small enough that a designer will do it over coffee. That is the difference between a habit and a study.
Sep 11, 2026Added to the Watching list from the prompt-evals brief and the 2026-09-11 vendor pass; the Chromatic review loop shapes the comparison and release entries.