Agency & Decisions

Release Decision

Solves: deciding whether a prompt change ships, with the evidence in view.

Watching — a candidate behavior we are tracking, not yet a published pattern.

watching emerging established contested fading 7 sightings since May 2026
Ask an assistant about this pattern Ask ChatGPT ↗Ask Claude ↗

The Release Decision attaches a test run to the moment a prompt change ships. The person sees the results beside the change. The check follows the review: a run with changes still awaiting a decision does not pass, and a denied change fails it. The team decides which regressions block and which only warn. The decision is recorded with the evidence that informed it, and accepted answers become the baselines for the next version.

When to Use It

  • Prompt changes reach real humans
  • A regression found after release is expensive
  • The team wants a record of what was known at ship time

When Not To

  • The change is internal and reversible in a minute
  • No eval set exists yet; write one first
  • The result would be ignored; fix that before adding a step

Three views

The exchange between the human, the agent and the system the agent acts on; who holds each part of it; and the component that implements it.

01 · the interaction
human agent system CI · model · release record attaches the eval set to the change runs the set against the new version in CI returns pass rate, regressions and cost the result against the threshold, in three words ships, blocks or overrides records the decision on the release
02 · who holds what
human interface agent Attached eval: The eval set and rubric tied to this release step 01 Attached eval Result: Pass, warn or block, with the regressions named and the unreviewed changes counted 02 Result Decision: The person ships, holds, or ships with a known regression 03 Decision Threshold: Which regressions block and which only warn, set by the team 04 Threshold Receipt: The decision and its evidence, recorded 05 Receipt
human interface agent Attached eval: The eval set and rubric tied to this release step 01 Attached eval Result: Pass, warn or block, with the regressions named and the unreviewed changes counted 02 Result Decision: The person ships, holds, or ships with a known regression 03 Decision Threshold: Which regressions block and which only warn, set by the team 04 Threshold Receipt: The decision and its evidence, recorded 05 Receipt
  1. 01 Attached eval The eval set and rubric tied to this release step
  2. 02 Result Pass, warn or block, with the regressions named and the unreviewed changes counted
  3. 03 Decision The person ships, holds, or ships with a known regression
  4. 04 Threshold Which regressions block and which only warn, set by the team
  5. 05 Receipt The decision and its evidence, recorded
03 · the component

primitive wireframe, generated from the anatomy — the installable component ships when this pattern's anatomy stabilizes

Field Notes

SEP 11 · Clipped · via Chromatic docs Comparison ViewRelease Decision

Chromatic solved the accept-or-deny review loop years ago

Swap snapshots for answers and the loop transfers to any agent that produces output a human reviews. The baseline moves only when a reviewer accepts a change, so no answer is approved twice and the build blocks until the review happens.

SEP 10 · Clipped · via Promptfoo on GitHub Release DecisionEval Run Setup

Promptfoo lets a pass percentage turn the release check red

A threshold is a decision made once, by whoever wrote the workflow file. The product does not ask whether this regression should block or only warn; that question needs a screen, not a setting.

JUN 3 · Clipped · via TheRouter (secondary) Release DecisionEval Run Setup

OpenAI retires its testing product and points builders elsewhere

A vendor retiring the surface moves the interaction into the team's own tools. That is an argument for patterns that outlive any one product, and for baselines the team owns.

MAY 25 · Clipped · via Langfuse changelog Comparison ViewProduction DriftRelease Decision

Langfuse puts the verdict on the release check in three words

Three words on the release check beat a dashboard that goes unopened. The open question is who decides: failing the job on a regression is a setting, and a person still owns the call.

Documented sightings (3) — anatomy-mapped observations from vendor materials
Promptfoo

The threshold is the decision: below the pass percentage, the check fails. Warn and block are not distinguished inside the product.

Promptfoo on GitHub · Sep 10, 2026
Langfuse

The pull request comment says passed, regressed, or failed to run; failing the job on a regression is a setting the team turns on.

Langfuse changelog · May 25, 2026
Chromatic

Adjacent field. The check follows the review: a build with unaccepted changes does not pass, and teams can require review before merge.

Chromatic docs · Sep 11, 2026

Side by Side

Anatomy part PromptfooLangfuseChromatic
Attached eval action in the workflowexperiment-actionUI Tests check on the pull request
Result pass/fail counts in a PR commentpassed, regressed, failed to runchanges pending, accepted, or denied
Decision merge or fixmerge or fixreviewer accepts or denies
Threshold fail-on-threshold pass percentageregression detection— (review, not a number)
Receipt PR comment with a viewer linkPR commentbaselines applied on merge

Tensions & Failure Modes

  • Block or warn: a hard block on each regression stops shipping; a warning on each regression gets ignored. The team sets which is which.
  • Whose call: an automated check can hold a release, and someone still owns the decision. Name the person.
  • Interaction or plumbing: the check runs in the deploy system, and the decision is a moment a person lives through. This pattern covers the moment.

The Story So Far

  1. Sep 11, 2026 Added to the Watching list from the prompt-evals brief and the 2026-09-11 vendor pass; the Chromatic review loop shapes the comparison and release entries.

Related Patterns