Understanding

Production Drift

Solves: knowing when quality slips in production, measured the same way as the tests.

Watching — a candidate behavior we are tracking, not yet a published pattern.

watching emerging established contested fading 7 sightings since Aug 2024
Ask an assistant about this pattern Ask ChatGPT ↗Ask Claude ↗

Production Drift measures live answers against the same rubric the tests used. The person sees quality over time, not only at release. A slip shows up as a change against the standard, with the cases behind it. Failures found here become new test cases.

When to Use It

  • The prompt runs in production at volume
  • Inputs change faster than the eval set
  • A slip would be felt by real humans before anyone looked

When Not To

  • Volume is too low to see a trend
  • Scoring would send private data to a model with no plan for it
  • No rubric exists; production scoring without a standard is a dashboard

Three views

The exchange between the human, the agent and the system the agent acts on; who holds each part of it; and the component that implements it.

01 · the interaction
human agent system production traffic · rubric samples production traffic on a schedule scores the sample with the rubric returns the trend against last week alerts when the trend crosses the line reads the sample; feeds a case back into the set
02 · who holds what
human interface agent Sample: Which live answers are scored, and how many 01 Sample Same rubric: The standard the tests used, applied to live traffic 02 Same rubric Trend: Quality over time, by dimension 03 Trend Alert: A slip past a threshold reaches the person 04 Alert Back to the set: A production failure becomes a test case 05 Back to the set
human interface agent Sample: Which live answers are scored, and how many 01 Sample Same rubric: The standard the tests used, applied to live traffic 02 Same rubric Trend: Quality over time, by dimension 03 Trend Alert: A slip past a threshold reaches the person 04 Alert Back to the set: A production failure becomes a test case 05 Back to the set
  1. 01 Sample Which live answers are scored, and how many
  2. 02 Same rubric The standard the tests used, applied to live traffic
  3. 03 Trend Quality over time, by dimension
  4. 04 Alert A slip past a threshold reaches the person
  5. 05 Back to the set A production failure becomes a test case
03 · the component

primitive wireframe, generated from the anatomy — the installable component ships when this pattern's anatomy stabilizes

Field Notes

SEP 18 · Paper · via arXiv ContextProduction DriftObservability

An RCT caught a deployed chat agent breaking its own written rules

The brief you write for an agent shapes its behavior only where that behavior can be checked. Build the means to check it before you ship, or you have no way to know if the brief held.

MAY 25 · Clipped · via Langfuse changelog Comparison ViewProduction DriftRelease Decision

Langfuse puts the verdict on the release check in three words

Three words on the release check beat a dashboard that goes unopened. The open question is who decides: failing the job on a regression is a setting, and a person still owns the call.

NOV 24 · Clipped · via Braintrust blog Production DriftEval Set Authoring

Braintrust turns a described failure into test cases and a scorer

Writing the test set was the chore that kept teams iterating by feel. Describing the failure in a sentence and getting rows and a scorer back is the step that turns the chore into a habit.

AUG 15 · Clipped · via Freeplay blog Case DrilldownProduction Drift

Freeplay asks the reviewer to agree or disagree, ten cases at a time

Ten at a time is the right unit. It is enough to calibrate a judge and small enough that a designer will do it over coffee. That is the difference between a habit and a study.

Documented sightings (3) — anatomy-mapped observations from vendor materials
Braintrust

Patterns runs Loop on a schedule over production logs and offers to make a scorer from a pattern it found.

Braintrust docs · Sep 11, 2026
Langfuse

A weighted rollout tracks metrics per prompt version while the new version takes a share of traffic.

Langfuse changelog · May 25, 2026
Freeplay

The same evaluation runs offline and on live traffic once its alignment score clears the bar.

Freeplay blog · Aug 15, 2024

Side by Side

Anatomy part BraintrustLangfuseFreeplay
Sample production logsrollout sharelive traffic
Same rubric same scorers onlinesame scoressame evaluation online and offline
Trend scores over timeper-version metricsmetric reviews
Alert
Back to the set pattern to scorer or datasettraces to datasetsreview queues to datasets

Tensions & Failure Modes

  • Two standards: if production is scored by a different rule than the tests, a slip in one is invisible in the other. One rubric, two places.
  • Sampling cost: scoring each live answer with a model is expensive. The sample must be enough to see a trend and no more.
  • Alert fatigue: a threshold set too tight pages humans for noise. Set it from the run-to-run variation the tests showed.

The Story So Far

  1. Sep 18, 2026 drift: quiet → too-early (30d sightings 0 → 1)
  2. Sep 11, 2026 Added to the Watching list from the prompt-evals brief and the 2026-09-11 vendor pass; the Chromatic review loop shapes the comparison and release entries.

Related Patterns