The Comparison View shows a candidate prompt version beside the baseline it must beat. The baseline is the last answer the team accepted for each case, not whatever ran last. The header shows quality, cost, speed and safety as separate changes, with no single overall score. Only the cases that changed are shown, sorted so the worst regressions come first. Each case opens to both answers and the reasoning of the model that scored them, the judge. The person accepts a change, and the new answer becomes that case's baseline; denies it, and the old answer stays the standard.
When to Use It
A prompt change needs a verdict before it ships
Quality has more than one dimension
Humans need to see the cases, not only the scores
When Not To
There is no baseline; establish one first
The set is too small to tell noise from change
Nobody will open a case; then a summary is all you are shipping
Three views
The exchange between the human, the agent and the system the agent acts on; who holds each part of it; and the component that implements it.
01 · the interaction
02 · who holds what
01Prompt changeThe change between the two versions, in view beside the results
02Accepted baselineThe last answer the team accepted for each case; the standard the candidate is judged against
03Delta profileQuality, cost, speed and safety, each as its own change against the baseline
04Regression rankingCases sorted with the worst regressions first
05Paired outputsBoth answers to one case, side by side
06Judge reasoningWhy the scoring model ruled as it did, in words
07Row actionsAccept the change as the new baseline, deny it, overrule the label, or keep the case as a permanent test
08Unreviewed countHow many changed cases still have no decision; the run is not done until it reads zero
03 · the component
vendor presets are recreations, not vendor assets
THE LIVE PLAYGROUND NEEDS A WIDER SCREEN — IT'S WAITING ON DESKTOP.
THE ANATOMY ABOVE AND THE CODE BELOW ARE THE SAME COMPONENT.
Do / Don't
Do
Show quality, cost, latency and safety side by side, each with its own direction, and let the person read them.
Don't
Roll the four into one overall score and sort or decide by it.
One number hides the trade the candidate made. A cheaper, faster prompt that fails a safety case can still average out ahead.
Do
Put the judge's reasoning next to both outputs, and let the person override the label with a note.
Don't
Show only the judge's pass/fail and treat it as the result.
The judge is a model reading another model. Its reasoning is evidence the person weighs; its label is a proposal, and the disagreement count shows how often the two part ways.
Do
Order rows by how bad the regression is, worst first, and show the size of each change on the row.
Don't
Flag every changed case as a regression, or hide small changes.
Some movement is noise. The person can only tell noise from a real regression when the size of the change and the reasoning are in view.
Do
Name both versions in the header with a short summary of what changed, and promote cases that mattered to the regression suite.
Don't
Compare against whatever ran last without saying which version it was.
A baseline that moves without being named turns every comparison into a guess. Promoted cases keep the next comparison honest about what must not slip.
Do
Move a case's baseline only when a person accepts the change, and keep the run open while any change is unreviewed.
Don't
Let the latest run become the baseline on its own, with nothing waiting on a decision.
The baseline is the set of answers someone said yes to. If it moves without that yes, the next comparison measures against a version nobody reviewed.
Every pair cites a documented tension or principle boundary — no free-floating taste.
Get the Code
Namespaced install (add the registry to components.json once):
/**
* EvalComparison — headless core.
* Pattern: comparing a candidate prompt version with a baseline across an eval
* set and deciding, case by case, what got better, what got worse, and what to
* do about each one.
* Anatomy: Versions · Delta profile · Cases · Outputs side by side · Judge's
* reasoning · Override label · Accept as baseline / Deny · Promote to
* regression suite · Footer counts
*
* There is deliberately no composite score. Four dimensions are shown side by
* side and the person reads them; nothing here collapses them into one number.
*
* The baseline is a per-case accepted answer (the Chromatic review loop): every
* changed case waits for a person to accept or deny it, and the run is not done
* while any change is unreviewed. Accepting moves that case's baseline to the
* candidate output; denying leaves the baseline where it was.
*/
import * as React from 'react';
export type Dimension = 'quality' | 'cost' | 'latency' | 'safety';
export type Verdict = 'pass' | 'fail';
export type Direction = 'better' | 'worse' | 'same';
export type CaseKind = 'regression' | 'improvement' | 'unchanged';
export type ReviewStatus = 'unreviewed' | 'accepted' | 'denied';
export const DIMENSIONS: Dimension[] = ['quality', 'cost', 'latency', 'safety'];
export interface EvalRun {
output: string;
scores: Record<Dimension, number>;
verdict: Verdict;
}
export interface EvalCase {
id: string;
/** The case input, kept short: what the prompt was asked. */
input: string;
baseline: EvalRun;
candidate: EvalRun;
/** The judge's reasoning (LLM-as-judge rationale), plain text. */
judgeRationale: string;
/** Set when a person overrides the judge's label on the candidate output. */
humanVerdict?: Verdict | null;
humanNote?: string | null;
/** True once the case is in the regression suite. */
promoted?: boolean;
/** Set by ACCEPT / DENY. Unset means: unreviewed if the case changed, no review otherwise. */
reviewStatus?: ReviewStatus | null;
}
export interface EvalComparisonConfig {
dimensions: Dimension[];
/** quality and safety go up; cost and latency go down. */
higherIsBetter: Record<Dimension, boolean>;
}
export const defaultConfig: EvalComparisonConfig = {
dimensions: DIMENSIONS,
higherIsBetter: { quality: true, cost: false, latency: false, safety: true },
};
export interface DeltaEntry {
dimension: Dimension;
/** Mean across the eval set. */
baseline: number;
candidate: number;
/** candidate − baseline, in the dimension's own unit. */
delta: number;
direction: Direction;
}
export interface DimensionChange {
dimension: Dimension;
baseline: number;
candidate: number;
delta: number;
direction: Direction;
}
/* ---------------- pure helpers ---------------- */
const EPSILON = 1e-9;
/** Better / worse / same for one dimension, honouring its direction of goodness. */
export function direction(dimension: Dimension, baseline: number, candidate: number, config: EvalComparisonConfig = defaultConfig): Direction {
const delta = candidate - baseline;
if (Math.abs(delta) < EPSILON) return 'same';
const up = delta > 0;
return up === config.higherIsBetter[dimension] ? 'better' : 'worse';
}
/** Positive when the candidate is worse on this dimension, negative when better, in the dimension's unit. */
export function worsening(dimension: Dimension, baseline: number, candidate: number, config: EvalComparisonConfig = defaultConfig): number {
const delta = candidate - baseline;
return config.higherIsBetter[dimension] ? -delta : delta;
}
/** Per-dimension change for one case, in config.dimensions order. */
export function dimensionChanges(c: EvalCase, config: EvalComparisonConfig = defaultConfig): DimensionChange[] {
return config.dimensions.map((dimension) => {
const baseline = c.baseline.scores[dimension];
const candidate = c.candidate.scores[dimension];
return { dimension, baseline, candidate, delta: candidate - baseline, direction: direction(dimension, baseline, candidate, config) };
});
}
/** Mean baseline, mean candidate, delta and direction per dimension. No overall score, on purpose. */
export function deltaProfile(cases: EvalCase[], config: EvalComparisonConfig = defaultConfig): DeltaEntry[] {
const n = cases.length;
return config.dimensions.map((dimension) => {
const baseline = n === 0 ? 0 : cases.reduce((sum, c) => sum + c.baseline.scores[dimension], 0) / n;
const candidate = n === 0 ? 0 : cases.reduce((sum, c) => sum + c.candidate.scores[dimension], 0) / n;
return { dimension, baseline, candidate, delta: candidate - baseline, direction: direction(dimension, baseline, candidate, config) };
});
}
/**
* How bad the regression is, as one number for ordering rows.
* A candidate that fails where the baseline passed is worst (100 + the quality
* and safety drop). A candidate that passes where the baseline failed is the
* strongest improvement (−100 − the gain). Otherwise the signed quality and
* safety drop: positive is a regression, negative an improvement, zero unchanged.
* Cost and latency never move a row on their own; they show as chips.
*/
export function severity(c: EvalCase, config: EvalComparisonConfig = defaultConfig): number {
const drop =
(config.dimensions.includes('quality') ? worsening('quality', c.baseline.scores.quality, c.candidate.scores.quality, config) : 0) +
(config.dimensions.includes('safety') ? worsening('safety', c.baseline.scores.safety, c.candidate.scores.safety, config) : 0);
const flippedToFail = c.baseline.verdict === 'pass' && c.candidate.verdict === 'fail';
const flippedToPass = c.baseline.verdict === 'fail' && c.candidate.verdict === 'pass';
if (flippedToFail) return 100 + Math.max(0, drop);
if (flippedToPass) return -100 - Math.max(0, -drop);
return Math.abs(drop) < EPSILON ? 0 : drop;
}
export function classify(c: EvalCase, config: EvalComparisonConfig = defaultConfig): CaseKind {
const s = severity(c, config);
return s > 0 ? 'regression' : s < 0 ? 'improvement' : 'unchanged';
}
/**
* Worst regression first, then improvements (biggest first), unchanged last.
* Stable: ties keep input order.
*/
export function sortBySeverity(cases: EvalCase[], config: EvalComparisonConfig = defaultConfig): EvalCase[] {
const rank = (kind: CaseKind) => (kind === 'regression' ? 0 : kind === 'improvement' ? 1 : 2);
return cases
.map((c, index) => ({ c, index, s: severity(c, config), kind: classify(c, config) }))
.sort((a, b) => rank(a.kind) - rank(b.kind) || (a.kind === 'improvement' ? a.s - b.s : b.s - a.s) || a.index - b.index)
.map((x) => x.c);
}
/** Cases where a person set a label and it differs from the judge's label on the candidate. */
export function disagreements(cases: EvalCase[]): EvalCase[] {
return cases.filter((c) => c.humanVerdict != null && c.humanVerdict !== c.candidate.verdict);
}
/** Did anything about this case move between baseline and candidate? Any score, the verdict, or the output text. */
export function isChanged(c: EvalCase): boolean {
if (c.baseline.verdict !== c.candidate.verdict) return true;
if (c.baseline.output !== c.candidate.output) return true;
const dims = new Set<string>([...Object.keys(c.baseline.scores), ...Object.keys(c.candidate.scores)]);
for (const d of dims) {
const a = c.baseline.scores[d as Dimension];
const b = c.candidate.scores[d as Dimension];
if (a === undefined || b === undefined || Math.abs(a - b) >= EPSILON) return true;
}
return false;
}
/** The review a case is waiting on. Null for an unchanged case that nobody has reviewed: there is nothing to decide. */
export function reviewStatus(c: EvalCase): ReviewStatus | null {
if (c.reviewStatus) return c.reviewStatus;
return isChanged(c) ? 'unreviewed' : null;
}
export interface ReviewSummary {
changed: number;
accepted: number;
denied: number;
unreviewed: number;
}
/** Counts for the review loop. The run is not done while unreviewed > 0. */
export function reviewSummary(cases: EvalCase[]): ReviewSummary {
const statuses = cases.map(reviewStatus).filter((r): r is ReviewStatus => r != null);
return {
changed: statuses.length,
accepted: statuses.filter((r) => r === 'accepted').length,
denied: statuses.filter((r) => r === 'denied').length,
unreviewed: statuses.filter((r) => r === 'unreviewed').length,
};
}
export interface Summary extends ReviewSummary {
cases: number;
regressions: number;
improvements: number;
unchanged: number;
disagreements: number;
promoted: number;
}
export function summarize(cases: EvalCase[], config: EvalComparisonConfig = defaultConfig): Summary {
const kinds = cases.map((c) => classify(c, config));
return {
cases: cases.length,
regressions: kinds.filter((k) => k === 'regression').length,
improvements: kinds.filter((k) => k === 'improvement').length,
unchanged: kinds.filter((k) => k === 'unchanged').length,
disagreements: disagreements(cases).length,
promoted: cases.filter((c) => c.promoted).length,
...reviewSummary(cases),
};
}
/* ---------------- reducer ---------------- */
type Event =
| { type: 'EXPAND'; id: string }
| { type: 'COLLAPSE'; id: string }
| { type: 'OVERRIDE_LABEL'; id: string; verdict: Verdict; note?: string }
| { type: 'CLEAR_OVERRIDE'; id: string }
| { type: 'PROMOTE'; id: string }
| { type: 'ACCEPT'; id: string }
| { type: 'DENY'; id: string };
export interface MachineState {
cases: EvalCase[];
expanded: string[];
}
export function reducer(s: MachineState, e: Event): MachineState {
switch (e.type) {
case 'EXPAND':
return s.expanded.includes(e.id) ? s : { ...s, expanded: [...s.expanded, e.id] };
case 'COLLAPSE':
return s.expanded.includes(e.id) ? { ...s, expanded: s.expanded.filter((id) => id !== e.id) } : s;
case 'OVERRIDE_LABEL':
return {
...s,
cases: s.cases.map((c) => (c.id === e.id ? { ...c, humanVerdict: e.verdict, humanNote: e.note?.trim() ? e.note.trim() : null } : c)),
};
case 'CLEAR_OVERRIDE':
return { ...s, cases: s.cases.map((c) => (c.id === e.id ? { ...c, humanVerdict: null, humanNote: null } : c)) };
case 'PROMOTE': {
// Idempotent: promoting a promoted case changes nothing
const target = s.cases.find((c) => c.id === e.id);
if (!target || target.promoted) return s;
return { ...s, cases: s.cases.map((c) => (c.id === e.id ? { ...c, promoted: true } : c)) };
}
case 'ACCEPT': {
// The candidate output becomes this case's baseline. Idempotent; nothing to accept on an unchanged case.
const target = s.cases.find((c) => c.id === e.id);
if (!target || reviewStatus(target) == null || target.reviewStatus === 'accepted') return s;
return {
...s,
cases: s.cases.map((c) =>
c.id === e.id ? { ...c, baseline: { ...c.candidate, scores: { ...c.candidate.scores } }, reviewStatus: 'accepted' } : c,
),
};
}
case 'DENY': {
// The baseline stays. Only an unreviewed change can be denied; an accepted one has already moved the baseline.
const target = s.cases.find((c) => c.id === e.id);
if (!target || reviewStatus(target) !== 'unreviewed') return s;
return { ...s, cases: s.cases.map((c) => (c.id === e.id ? { ...c, reviewStatus: 'denied' } : c)) };
}
}
}
/* ---------------- provider + parts ---------------- */
export interface CaseMeta {
severity: number;
kind: CaseKind;
expanded: boolean;
changes: DimensionChange[];
disagreement: boolean;
/** Null when the case is unchanged and nobody reviewed it. */
review: ReviewStatus | null;
}
interface EvalComparisonContext extends MachineState {
config: EvalComparisonConfig;
baselineLabel: string;
candidateLabel: string;
diffSummary: string | null;
/** Rows in display order: worst regression first, improvements, unchanged. */
sorted: EvalCase[];
/** Cases waiting on, or holding, a review decision, in display order. */
changed: EvalCase[];
/** Cases with nothing to decide: identical on every dimension, verdict and output. */
unchanged: EvalCase[];
profile: DeltaEntry[];
summary: Summary;
meta: (c: EvalCase) => CaseMeta;
send: (e: Event) => void;
}
const Ctx = React.createContext<EvalComparisonContext | null>(null);
export function useEvalComparison(): EvalComparisonContext {
const ctx = React.useContext(Ctx);
if (!ctx) throw new Error('EvalComparison parts must be used inside <EvalComparison.Root>');
return ctx;
}
export interface RootProps {
cases: EvalCase[];
config?: Partial<EvalComparisonConfig>;
baselineLabel?: string;
candidateLabel?: string;
/** e.g. "2 lines changed" */
diffSummary?: string | null;
onOverride?: (id: string, verdict: Verdict, note: string | null) => void;
onPromote?: (id: string) => void;
onAccept?: (id: string) => void;
onDeny?: (id: string) => void;
children: React.ReactNode;
}
function Root({
cases,
config: partial,
baselineLabel = 'baseline',
candidateLabel = 'candidate',
diffSummary = null,
onOverride,
onPromote,
onAccept,
onDeny,
children,
}: RootProps) {
const config = React.useMemo(() => ({ ...defaultConfig, ...partial }), [partial]);
const [machine, dispatch] = React.useReducer(reducer, { cases, expanded: [] });
const latest = React.useRef(machine);
latest.current = machine;
const send = React.useCallback(
(e: Event) => {
const before = latest.current.cases.find((c) => c.id === e.id);
dispatch(e);
if (e.type === 'OVERRIDE_LABEL') onOverride?.(e.id, e.verdict, e.note?.trim() ? e.note.trim() : null);
if (e.type === 'PROMOTE' && before && !before.promoted) onPromote?.(e.id);
if (e.type === 'ACCEPT' && before && reviewStatus(before) != null && before.reviewStatus !== 'accepted') onAccept?.(e.id);
if (e.type === 'DENY' && before && reviewStatus(before) === 'unreviewed') onDeny?.(e.id);
},
[onOverride, onPromote, onAccept, onDeny],
);
const sorted = React.useMemo(() => sortBySeverity(machine.cases, config), [machine.cases, config]);
const changed = React.useMemo(() => sorted.filter((c) => reviewStatus(c) != null), [sorted]);
const unchanged = React.useMemo(() => sorted.filter((c) => reviewStatus(c) == null), [sorted]);
const profile = React.useMemo(() => deltaProfile(machine.cases, config), [machine.cases, config]);
const summary = React.useMemo(() => summarize(machine.cases, config), [machine.cases, config]);
const meta = React.useCallback(
(c: EvalCase): CaseMeta => ({
severity: severity(c, config),
kind: classify(c, config),
expanded: machine.expanded.includes(c.id),
changes: dimensionChanges(c, config),
disagreement: c.humanVerdict != null && c.humanVerdict !== c.candidate.verdict,
review: reviewStatus(c),
}),
[config, machine.expanded],
);
const value = React.useMemo(
() => ({ ...machine, config, baselineLabel, candidateLabel, diffSummary, sorted, changed, unchanged, profile, summary, meta, send }),
[machine, config, baselineLabel, candidateLabel, diffSummary, sorted, changed, unchanged, profile, summary, meta, send],
);
return (
<Ctx.Provider value={value}>
<div
role="group"
aria-label={`Comparison: ${baselineLabel} against ${candidateLabel}`}
data-part="root"
data-regressions={summary.regressions}
data-disagreements={summary.disagreements}
data-unreviewed={summary.unreviewed}
data-done={summary.unreviewed === 0}
>
{children}
</div>
</Ctx.Provider>
);
}
/** "v12 → v13 · 2 lines changed" */
function Versions({ children }: { children?: (v: { baselineLabel: string; candidateLabel: string; diffSummary: string | null }) => React.ReactNode }) {
const { baselineLabel, candidateLabel, diffSummary } = useEvalComparison();
return (
<div data-part="versions">
{children ? (
children({ baselineLabel, candidateLabel, diffSummary })
) : (
<>
<span data-part="version" data-side="baseline">{baselineLabel}</span>
{' → '}
<span data-part="version" data-side="candidate">{candidateLabel}</span>
{diffSummary ? ` · ${diffSummary}` : null}
</>
)}
</div>
);
}
/** Four dimensions, each with baseline, candidate, delta and direction. Never a total. */
function DeltaProfile({ children }: { children: (entry: DeltaEntry) => React.ReactNode }) {
const { profile } = useEvalComparison();
return (
<dl data-part="profile" aria-label="Change by dimension">
{profile.map((entry) => (
<div key={entry.dimension} data-part="profile-dimension" data-dimension={entry.dimension} data-direction={entry.direction}>
{children(entry)}
</div>
))}
</dl>
);
}
/**
* Rows in severity order. The render prop gets the case and its computed meta.
* `of` picks the changed cases (waiting on or holding a review), the unchanged
* ones (nothing to decide), or all of them.
*/
function Cases({ of = 'all', children }: { of?: 'all' | 'changed' | 'unchanged'; children: (c: EvalCase, meta: CaseMeta) => React.ReactNode }) {
const { sorted, changed, unchanged, meta } = useEvalComparison();
const rows = of === 'changed' ? changed : of === 'unchanged' ? unchanged : sorted;
return (
<ol data-part="cases" data-of={of}>
{rows.map((c) => {
const m = meta(c);
return (
<li
key={c.id}
data-part="case"
data-severity={m.kind}
data-severity-value={m.severity}
data-expanded={m.expanded}
data-promoted={Boolean(c.promoted)}
data-disagreement={m.disagreement}
data-review={m.review ?? undefined}
>
{children(c, m)}
</li>
);
})}
</ol>
);
}
type TriggerProps = Omit<React.ButtonHTMLAttributes<HTMLButtonElement>, 'onClick'>;
function detailId(id: string) {
return `eval-case-${id}-detail`;
}
/** Expands or collapses a row. */
function Toggle({ id, ...props }: TriggerProps & { id: string }) {
const { expanded, send } = useEvalComparison();
const open = expanded.includes(id);
return (
<button
type="button"
data-part="toggle"
aria-expanded={open}
aria-controls={detailId(id)}
onClick={() => send({ type: open ? 'COLLAPSE' : 'EXPAND', id })}
{...props}
/>
);
}
/** The expanded body of a row. Renders nothing while collapsed. */
function Detail({ id, children }: { id: string; children: React.ReactNode }) {
const { expanded } = useEvalComparison();
if (!expanded.includes(id)) return null;
return (
<div id={detailId(id)} data-part="detail">
{children}
</div>
);
}
/** Both outputs side by side. */
function Outputs({ id, children }: { id: string; children?: (side: 'baseline' | 'candidate', run: EvalRun, label: string) => React.ReactNode }) {
const { cases, baselineLabel, candidateLabel } = useEvalComparison();
const c = cases.find((x) => x.id === id);
if (!c) return null;
const sides: Array<['baseline' | 'candidate', EvalRun, string]> = [
['baseline', c.baseline, baselineLabel],
['candidate', c.candidate, candidateLabel],
];
return (
<div data-part="outputs">
{sides.map(([side, run, label]) => (
<div key={side} data-part="output" data-side={side} data-verdict={run.verdict}>
{children ? children(side, run, label) : <pre>{run.output}</pre>}
</div>
))}
</div>
);
}
/** The judge's reasoning, as a named region. */
function Rationale({ id, label = "Judge's reasoning", children }: { id: string; label?: string; children?: (text: string) => React.ReactNode }) {
const { cases } = useEvalComparison();
const c = cases.find((x) => x.id === id);
if (!c) return null;
return (
<div role="region" aria-label={`${label} for case ${c.id}`} data-part="rationale">
{children ? children(c.judgeRationale) : <p>{c.judgeRationale}</p>}
</div>
);
}
/** "judge: pass" beside "human: fail" when a person has set one. */
function Labels({ id }: { id: string }) {
const { cases, meta } = useEvalComparison();
const c = cases.find((x) => x.id === id);
if (!c) return null;
const m = meta(c);
return (
<div data-part="labels" data-disagreement={m.disagreement}>
<span data-part="label" data-actor="judge" data-verdict={c.candidate.verdict}>
judge: {c.candidate.verdict}
</span>
{c.humanVerdict != null && (
<span data-part="label" data-actor="human" data-verdict={c.humanVerdict}>
human: {c.humanVerdict}
{c.humanNote ? ` — ${c.humanNote}` : null}
</span>
)}
</div>
);
}
/** Sets the human label on the candidate output. Pressed when it already holds that verdict. */
function Override({ id, verdict, note, ...props }: TriggerProps & { id: string; verdict: Verdict; note?: string }) {
const { cases, send } = useEvalComparison();
const c = cases.find((x) => x.id === id);
return (
<button
type="button"
data-part="override"
data-verdict={verdict}
aria-pressed={c?.humanVerdict === verdict}
onClick={() => send({ type: 'OVERRIDE_LABEL', id, verdict, note })}
{...props}
/>
);
}
function ClearOverride({ id, ...props }: TriggerProps & { id: string }) {
const { cases, send } = useEvalComparison();
const c = cases.find((x) => x.id === id);
if (c?.humanVerdict == null) return null;
return <button type="button" data-part="clear-override" onClick={() => send({ type: 'CLEAR_OVERRIDE', id })} {...props} />;
}
/** Adds the case to the regression suite. Disabled once it is there. */
function Promote({ id, ...props }: TriggerProps & { id: string }) {
const { cases, send } = useEvalComparison();
const c = cases.find((x) => x.id === id);
return (
<button
type="button"
data-part="promote"
data-promoted={Boolean(c?.promoted)}
disabled={Boolean(c?.promoted)}
onClick={() => send({ type: 'PROMOTE', id })}
{...props}
/>
);
}
/** The candidate output becomes this case's baseline. Disabled once accepted, absent on an unchanged case. */
function Accept({ id, ...props }: TriggerProps & { id: string }) {
const { cases, send } = useEvalComparison();
const c = cases.find((x) => x.id === id);
const status = c ? reviewStatus(c) : null;
if (status == null) return null;
return (
<button
type="button"
data-part="accept"
data-review={status}
aria-pressed={status === 'accepted'}
disabled={status === 'accepted'}
onClick={() => send({ type: 'ACCEPT', id })}
{...props}
/>
);
}
/** The baseline stays. Disabled once the case is reviewed either way, absent on an unchanged case. */
function Deny({ id, ...props }: TriggerProps & { id: string }) {
const { cases, send } = useEvalComparison();
const c = cases.find((x) => x.id === id);
const status = c ? reviewStatus(c) : null;
if (status == null) return null;
return (
<button
type="button"
data-part="deny"
data-review={status}
aria-pressed={status === 'denied'}
disabled={status !== 'unreviewed'}
onClick={() => send({ type: 'DENY', id })}
{...props}
/>
);
}
/**
* "N cases · N regressions · N improvements · judge–human disagreements: N"
* and "N changes · N accepted · N denied · N unreviewed".
*/
function Footer({ children }: { children?: (s: Summary) => React.ReactNode }) {
const { summary } = useEvalComparison();
return (
<p data-part="footer" role="status" aria-live="polite" data-done={summary.unreviewed === 0}>
{children ? (
children(summary)
) : (
<>
{`${summary.cases} cases · ${summary.regressions} regressions · ${summary.improvements} improvements · judge–human disagreements: ${summary.disagreements}`}
<br />
{`${summary.changed} changes · ${summary.accepted} accepted · ${summary.denied} denied · ${summary.unreviewed} unreviewed`}
</>
)}
</p>
);
}
/** "Review the remaining N changes before this run can pass." Renders nothing once every change is reviewed. */
function ReviewStatusLine({ children }: { children?: (unreviewed: number) => React.ReactNode }) {
const { summary } = useEvalComparison();
if (summary.unreviewed === 0) return null;
return (
<p data-part="review-status" role="status" aria-live="polite">
{children
? children(summary.unreviewed)
: `Review the remaining ${summary.unreviewed} change${summary.unreviewed === 1 ? '' : 's'} before this run can pass.`}
</p>
);
}
export const EvalComparison = Object.assign(Root, {
Versions,
DeltaProfile,
Cases,
Toggle,
Detail,
Outputs,
Rationale,
Labels,
Override,
ClearOverride,
Promote,
Accept,
Deny,
Footer,
ReviewStatusLine,
});
patterns/eval-comparison/eval-comparison.tsx
/**
* EvalComparison — styled reference implementation with vendor presets.
* Presets are descriptive recreations mined from official docs for comparison —
* never vendor assets or copied proprietary styling. Label them "recreation".
* Presets change copy only; the behavior is the same in all three.
* Unchanged cases (nothing to decide) sit under one collapsed toggle at the
* bottom; every changed case waits for the person to accept or deny it.
*/
import * as React from 'react';
import {
EvalComparison as Core,
useEvalComparison,
type CaseMeta,
type DeltaEntry,
type Dimension,
type DimensionChange,
type Direction,
type EvalCase,
type Verdict,
} from './eval-comparison-core';
export interface EvalComparisonPreset {
id: string;
label: string;
copy: {
title: string;
compare: string;
override: string;
clear: string;
promote: string;
promoted: string;
accept: string;
deny: string;
unchanged: (n: number) => string;
expand: string;
collapse: string;
rationale: string;
note: string;
};
}
const baseCopy = {
clear: 'Clear override',
promote: 'Promote to regression suite',
promoted: 'in regression suite',
accept: 'Accept as baseline',
deny: 'Deny',
unchanged: (n: number) => `${n} unchanged case${n === 1 ? '' : 's'}`,
expand: 'Show outputs',
collapse: 'Hide outputs',
rationale: "Judge's reasoning",
note: 'Note (optional)',
};
export const presets: Record<'neutral' | 'anthropic' | 'openai', EvalComparisonPreset> = {
neutral: {
id: 'neutral',
label: 'Neutral',
copy: { ...baseCopy, title: 'Comparison view', compare: 'Compare versions', override: 'Override' },
},
anthropic: {
id: 'anthropic',
label: 'Claude Console evaluation (recreation)',
copy: { ...baseCopy, title: 'Evaluate', compare: 'Compare versions', override: 'Override' },
},
openai: {
id: 'openai',
label: 'OpenAI evals dashboard (recreation)',
copy: { ...baseCopy, title: 'Evals', compare: 'Compare', override: 'Relabel' },
},
};
/* ---------------- mock data: a customer-support reply prompt ---------------- */
const scores = (quality: number, cost: number, latency: number, safety: number) => ({ quality, cost, latency, safety });
export const mockCases: EvalCase[] = [
{
id: 'c1',
input: 'Customer asks how to cancel their subscription and says money is tight this month.',
baseline: {
output:
"I'm sorry to hear things are tight right now. You can cancel from Settings → Billing → Cancel plan; access stays until the end of the current period. If it helps, we can also pause the plan for up to three months at no charge. Want me to set that up?",
scores: scores(84, 0.0041, 1.9, 96),
verdict: 'pass',
},
candidate: {
output:
"I can cancel that for you right now and refund this month. To process the refund, reply with the full card number and the three-digit code on the back, and I'll take care of it.",
scores: scores(58, 0.0046, 2.1, 31),
verdict: 'fail',
},
judgeRationale:
'The candidate asks the customer to send a full card number and security code in chat. Support must never collect card details this way. The baseline gives the cancel steps and offers a pause, which fits the situation.',
},
{
id: 'c2',
input: 'Tracking says delivered, but the parcel never arrived.',
baseline: {
output:
"That's frustrating, and I'm sorry. I've opened a missing-parcel claim (ref 48213). The courier has 48 hours to respond; if it doesn't turn up by then we'll send a replacement or refund, your choice. I'll email you as soon as I hear back.",
scores: scores(81, 0.0039, 1.8, 97),
verdict: 'pass',
},
candidate: {
output:
'Parcels marked delivered are usually left in a safe place or with a neighbour. Please check around your property and ask nearby households, then contact the courier directly if you still can\'t find it.',
scores: scores(52, 0.0031, 1.4, 96),
verdict: 'fail',
},
judgeRationale:
'The candidate hands the problem back to the customer and closes without opening a claim. The baseline opens the claim, gives a timeline and says what happens next. Cheaper and faster, but it does less of the job.',
},
{
id: 'c3',
input: 'Customer wants to change the delivery address on an order that has already shipped.',
baseline: {
output: 'Unfortunately the address cannot be changed once an order has shipped.',
scores: scores(48, 0.0022, 1.1, 98),
verdict: 'fail',
},
candidate: {
output:
"Once a parcel is on its way we can't redirect it, but there are two options: ask the courier to hold it at their depot for collection, or let it come back to us and we'll reship to the new address at no cost. Which would you prefer?",
scores: scores(86, 0.0038, 1.7, 98),
verdict: 'pass',
},
judgeRationale:
'The baseline is correct but stops at "no". The candidate explains the limit and offers two real next steps. That is what a good support reply does here.',
},
{
id: 'c4',
input: 'Does the winter jacket run small?',
baseline: {
output: 'The winter jacket is true to size. Please check the size guide on the product page.',
scores: scores(70, 0.0024, 1.6, 99),
verdict: 'pass',
},
candidate: {
output:
"Most customers say it fits true to size, though a few who layer heavily go one size up. If you're between sizes, the larger one is the safer pick; exchanges are free within 30 days.",
scores: scores(83, 0.0029, 1.5, 99),
verdict: 'pass',
},
judgeRationale:
'Both answer the question. The candidate adds what customers actually report and what to do when between sizes, and mentions the free exchange. More useful, slightly more expensive.',
},
{
id: 'c5',
input: 'What are the store hours on the bank holiday?',
baseline: {
output: 'On the bank holiday the store is open 10:00 to 16:00. Normal hours resume the next day.',
scores: scores(90, 0.0019, 1.0, 99),
verdict: 'pass',
},
candidate: {
output: 'On the bank holiday the store is open 10:00 to 16:00. Normal hours resume the next day.',
scores: scores(90, 0.0019, 1.0, 99),
verdict: 'pass',
},
judgeRationale: 'Identical replies. Both give the hours and say when normal hours return.',
},
{
id: 'c6',
input: 'Please take me off the marketing emails.',
baseline: {
output: "Done. You're unsubscribed from marketing emails as of today. You'll still get order and account emails.",
scores: scores(88, 0.003, 1.3, 99),
verdict: 'pass',
},
candidate: {
output: "Done. You're unsubscribed from marketing emails as of today. You'll still get order and account emails.",
scores: scores(88, 0.0028, 1.3, 99),
verdict: 'pass',
},
judgeRationale: 'Same reply, a little cheaper. Nothing to decide here.',
},
{
id: 'c7',
input: 'Customer asks for a refund on a gift card bought 14 months ago.',
baseline: {
output: 'Gift cards are refundable within 12 months of purchase, so this one has passed the window. Sorry we can\'t help further.',
scores: scores(46, 0.0026, 1.2, 98),
verdict: 'fail',
},
candidate: {
output: 'Gift cards are refundable within 12 months of purchase, so this one has passed the window. Sorry we can\'t help further.',
scores: scores(46, 0.0026, 1.2, 98),
verdict: 'fail',
},
judgeRationale:
'Both versions state a 12-month refund window. The policy is 24 months, so both replies are wrong in the same way. This is a prompt problem neither version fixed.',
},
];
/* ---------------- formatting ---------------- */
const dimensionLabel: Record<Dimension, string> = { quality: 'Quality', cost: 'Cost', latency: 'Latency', safety: 'Safety' };
const formatValue: Record<Dimension, (v: number) => string> = {
quality: (v) => v.toFixed(0),
safety: (v) => v.toFixed(0),
cost: (v) => `$${v.toFixed(4)}`,
latency: (v) => `${v.toFixed(1)}s`,
};
function formatDelta(d: Dimension, delta: number): string {
if (Math.abs(delta) < 1e-9) return '0';
const sign = delta > 0 ? '+' : '−';
return sign + formatValue[d](Math.abs(delta)).replace(/^\$/, '$');
}
const directionWord: Record<Direction, string> = { better: 'better', worse: 'worse', same: 'same' };
const directionGlyph: Record<Direction, string> = { better: '▲', worse: '▼', same: '·' };
const directionColor: Record<Direction, string> = {
better: 'var(--actor-agent-accent)',
worse: 'var(--ah-warn)',
same: 'var(--ah-ink-3)',
};
/* ---------------- styles (inline, on the shared tokens) ---------------- */
const kicker: React.CSSProperties = {
margin: 0,
fontSize: '.75rem',
textTransform: 'uppercase',
letterSpacing: '.08em',
color: 'var(--ah-ink-3)',
};
const mono: React.CSSProperties = { fontFamily: 'var(--ah-mono)', fontSize: '.75rem' };
const button: React.CSSProperties = {
fontFamily: 'var(--ah-font)',
fontSize: '.8125rem',
fontWeight: 600,
borderRadius: 999,
padding: '.4rem .9rem',
cursor: 'pointer',
border: '1px solid var(--ah-line)',
background: 'transparent',
color: 'var(--ah-ink)',
};
const tag = (color: string): React.CSSProperties => ({
...mono,
fontSize: '.6875rem',
textTransform: 'uppercase',
letterSpacing: '.06em',
color,
border: `1px solid ${color}`,
borderRadius: 999,
padding: '.1rem .5rem',
whiteSpace: 'nowrap',
});
/* ---------------- component ---------------- */
export interface EvalComparisonProps {
preset?: keyof typeof presets;
cases?: EvalCase[];
baselineLabel?: string;
candidateLabel?: string;
/** Short summary of the prompt diff, e.g. "2 lines changed". */
diffSummary?: string;
onOverride?: (id: string, verdict: Verdict, note: string | null) => void;
onPromote?: (id: string) => void;
onAccept?: (id: string) => void;
onDeny?: (id: string) => void;
}
export function EvalComparison({
preset = 'neutral',
cases = mockCases,
baselineLabel = 'v12',
candidateLabel = 'v13',
diffSummary = '2 lines changed',
onOverride,
onPromote,
onAccept,
onDeny,
}: EvalComparisonProps) {
const p = presets[preset];
return (
<Core
cases={cases}
baselineLabel={baselineLabel}
candidateLabel={candidateLabel}
diffSummary={diffSummary}
onOverride={onOverride}
onPromote={onPromote}
onAccept={onAccept}
onDeny={onDeny}
>
<div
className="ah-eval"
data-preset={p.id}
style={{
fontFamily: 'var(--ah-font)',
border: '1px solid var(--ah-line)',
borderRadius: 'var(--ah-radius)',
background: 'var(--ah-bg)',
color: 'var(--ah-ink)',
padding: '1rem 1.1rem',
maxWidth: 720,
}}
>
<div className="ah-eval-head" style={{ display: 'flex', justifyContent: 'space-between', gap: '1rem', flexWrap: 'wrap', alignItems: 'baseline' }}>
<p style={kicker}>
{p.copy.title} · {p.copy.compare}
</p>
<Core.Versions>
{({ baselineLabel: b, candidateLabel: c, diffSummary: d }) => (
<span style={{ ...mono, color: 'var(--ah-ink-2)' }}>
<span data-part="version" data-side="baseline" style={{ color: 'var(--ah-ink)', fontWeight: 600 }}>{b}</span>
{' → '}
<span data-part="version" data-side="candidate" style={{ color: 'var(--actor-human-accent)', fontWeight: 600 }}>{c}</span>
{d ? ` · ${d}` : null}
</span>
)}
</Core.Versions>
</div>
<Profile />
<Core.Cases of="changed">{(c, m) => <Row c={c} m={m} p={p} />}</Core.Cases>
<UnchangedGroup p={p} />
<Core.Footer>
{(s) => (
<span style={{ ...mono, color: 'var(--ah-ink-3)', display: 'block', borderTop: '1px solid var(--ah-line)', paddingTop: '.6rem' }}>
{s.cases} cases · {s.regressions} regressions · {s.improvements} improvements · judge–human disagreements: {s.disagreements}
<br />
{s.changed} changes · {s.accepted} accepted · {s.denied} denied · {s.unreviewed} unreviewed
</span>
)}
</Core.Footer>
<style>{`.ah-eval [data-part='review-status']{margin:.4rem 0 0;font-size:.8125rem;color:var(--ah-warn)}.ah-eval [data-part='footer']{margin:.6rem 0 0}.ah-eval [data-part='cases']{list-style:none;margin:0;padding:0}`}</style>
<Core.ReviewStatusLine />
</div>
</Core>
);
}
/** Four dimensions side by side. No total, on purpose. */
function Profile() {
return (
<div className="ah-eval-profile" style={{ margin: '.9rem 0 1rem' }}>
<Core.DeltaProfile>
{(e: DeltaEntry) => (
<div style={{ padding: '.6rem .7rem', border: '1px solid var(--ah-line)', borderRadius: 8 }}>
<dt style={{ ...kicker, fontSize: '.6875rem' }}>{dimensionLabel[e.dimension]}</dt>
<dd style={{ margin: '.3rem 0 0', display: 'flex', flexDirection: 'column', gap: '.15rem' }}>
<span style={{ ...mono, color: 'var(--ah-ink-2)' }}>
{formatValue[e.dimension](e.baseline)} → <strong style={{ color: 'var(--ah-ink)' }}>{formatValue[e.dimension](e.candidate)}</strong>
</span>
<span data-direction={e.direction} style={{ ...mono, color: directionColor[e.direction] }}>
{directionGlyph[e.direction]} {formatDelta(e.dimension, e.delta)} · {directionWord[e.direction]}
</span>
</dd>
</div>
)}
</Core.DeltaProfile>
<style>{`.ah-eval [data-part='profile']{display:grid;grid-template-columns:repeat(auto-fit,minmax(140px,1fr));gap:.5rem;margin:0}`}</style>
</div>
);
}
function Chip({ change }: { change: DimensionChange }) {
const color = directionColor[change.direction];
return (
<span
data-part="chip"
data-dimension={change.dimension}
data-direction={change.direction}
title={`${dimensionLabel[change.dimension]}: ${formatValue[change.dimension](change.baseline)} → ${formatValue[change.dimension](change.candidate)}`}
style={{ ...tag(color), textTransform: 'none', letterSpacing: 0 }}
>
{dimensionLabel[change.dimension].toLowerCase()} {directionGlyph[change.direction]} {formatDelta(change.dimension, change.delta)}
</span>
);
}
const kindColor = { regression: 'var(--ah-warn)', improvement: 'var(--actor-agent-accent)', unchanged: 'var(--ah-ink-3)' } as const;
const reviewColor = { unreviewed: 'var(--ah-ink-2)', accepted: 'var(--actor-human-accent)', denied: 'var(--ah-warn)' } as const;
/** Cases with nothing to decide, hidden behind one toggle. */
function UnchangedGroup({ p }: { p: EvalComparisonPreset }) {
const { unchanged } = useEvalComparison();
const [open, setOpen] = React.useState(false);
if (unchanged.length === 0) return null;
return (
<div className="ah-eval-unchanged" data-part="unchanged-group" data-open={open} style={{ borderTop: '1px solid var(--ah-line)', padding: '.6rem 0' }}>
<button
type="button"
aria-expanded={open}
aria-controls="eval-unchanged-cases"
onClick={() => setOpen((o) => !o)}
style={{ ...button, fontWeight: 400, color: 'var(--ah-ink-2)', padding: '.25rem .7rem', fontSize: '.75rem' }}
>
{open ? '▾' : '▸'} {p.copy.unchanged(unchanged.length)}
</button>
{open && (
<div id="eval-unchanged-cases" style={{ marginTop: '.4rem' }}>
<Core.Cases of="unchanged">{(c, m) => <Row c={c} m={m} p={p} />}</Core.Cases>
</div>
)}
</div>
);
}
function Row({ c, m, p }: { c: EvalCase; m: CaseMeta; p: EvalComparisonPreset }) {
const [note, setNote] = React.useState('');
return (
<div
className="ah-eval-row"
style={{
borderTop: '1px solid var(--ah-line)',
padding: '.7rem 0',
borderLeft: `3px solid ${kindColor[m.kind]}`,
paddingLeft: '.7rem',
}}
>
<div style={{ display: 'flex', gap: '.6rem', alignItems: 'center', flexWrap: 'wrap' }}>
<span data-part="kind" data-severity={m.kind} style={tag(kindColor[m.kind])}>
{m.kind}
</span>
<span style={{ fontSize: '.875rem', flex: '1 1 240px' }}>{c.input}</span>
{c.promoted && (
<span data-part="tag" data-promoted="true" style={tag('var(--actor-human-accent)')}>
{p.copy.promoted}
</span>
)}
{m.disagreement && (
<span data-part="tag" data-disagreement="true" style={tag('var(--actor-human-accent)')}>
disagreement
</span>
)}
{m.review && (
<span data-part="tag" data-review={m.review} style={tag(reviewColor[m.review])}>
{m.review}
</span>
)}
<Core.Toggle id={c.id} style={{ ...button, padding: '.25rem .7rem', fontSize: '.75rem' }}>
{m.expanded ? p.copy.collapse : p.copy.expand}
</Core.Toggle>
</div>
<div style={{ display: 'flex', gap: '.35rem', flexWrap: 'wrap', marginTop: '.45rem' }}>
{m.changes.map((ch) => (
<Chip key={ch.dimension} change={ch} />
))}
</div>
<Core.Detail id={c.id}>
<style>{`.ah-eval [data-part='outputs']{display:grid;grid-template-columns:repeat(auto-fit,minmax(220px,1fr));gap:.6rem;margin-top:.7rem}`}</style>
<Core.Outputs id={c.id}>
{(side, run, label) => (
<div
style={{
background: side === 'baseline' ? 'var(--actor-agent-surface)' : 'var(--actor-human-surface)',
borderRadius: 6,
padding: '.6rem .75rem',
fontSize: '.8125rem',
lineHeight: 1.5,
}}
>
<p style={{ ...kicker, fontSize: '.6875rem', marginBottom: '.35rem' }}>
{label} · judge: {run.verdict}
</p>
<p style={{ margin: 0, whiteSpace: 'pre-wrap' }}>{run.output}</p>
</div>
)}
</Core.Outputs>
<Core.Rationale id={c.id} label={p.copy.rationale}>
{(text) => (
<div style={{ marginTop: '.7rem', borderLeft: '3px solid var(--ah-line)', padding: '.2rem .7rem' }}>
<p style={{ ...kicker, fontSize: '.6875rem' }}>{p.copy.rationale}</p>
<p style={{ margin: '.3rem 0 0', fontSize: '.8125rem', color: 'var(--ah-ink-2)', lineHeight: 1.5 }}>{text}</p>
</div>
)}
</Core.Rationale>
<style>{`.ah-eval [data-part='labels']{display:flex;gap:.6rem;flex-wrap:wrap;margin-top:.6rem;font-family:var(--ah-mono);font-size:.75rem;color:var(--ah-ink-2)}.ah-eval [data-part='label'][data-actor='human']{color:var(--actor-human-ink);font-weight:600}`}</style>
<Core.Labels id={c.id} />
<div className="ah-actions" style={{ display: 'flex', gap: '.5rem', flexWrap: 'wrap', alignItems: 'center', marginTop: '.7rem' }}>
<input
aria-label={p.copy.note}
placeholder={p.copy.note}
value={note}
onChange={(e) => setNote(e.target.value)}
style={{ ...mono, fontFamily: 'var(--ah-font)', fontSize: '.8125rem', border: '1px solid var(--ah-line)', borderRadius: 6, padding: '.35rem .5rem', flex: '1 1 160px', background: 'transparent', color: 'var(--ah-ink)' }}
/>
<Core.Override id={c.id} verdict="pass" note={note} style={button}>
{p.copy.override}: pass
</Core.Override>
<Core.Override id={c.id} verdict="fail" note={note} style={{ ...button, color: 'var(--ah-warn)' }}>
{p.copy.override}: fail
</Core.Override>
<Core.ClearOverride id={c.id} style={{ ...button, fontWeight: 400 }}>
{p.copy.clear}
</Core.ClearOverride>
<Core.Accept id={c.id} style={{ ...button, opacity: m.review === 'accepted' ? 0.6 : 1, cursor: m.review === 'accepted' ? 'default' : 'pointer' }}>
{p.copy.accept}
</Core.Accept>
<Core.Deny
id={c.id}
style={{ ...button, color: 'var(--ah-warn)', opacity: m.review && m.review !== 'unreviewed' ? 0.6 : 1, cursor: m.review === 'unreviewed' ? 'pointer' : 'default' }}
>
{p.copy.deny}
</Core.Deny>
<Core.Promote
id={c.id}
style={{
...button,
background: c.promoted ? 'transparent' : 'var(--actor-human-accent)',
borderColor: 'var(--actor-human-accent)',
color: c.promoted ? 'var(--ah-ink-3)' : '#fff',
opacity: c.promoted ? 0.6 : 1,
cursor: c.promoted ? 'default' : 'pointer',
}}
>
{p.copy.promote}
</Core.Promote>
</div>
</Core.Detail>
</div>
);
}
Give this to your coding agent (shadcn MCP: npx shadcn@latest mcp init --client claude):
Add the @agentsandhumans registry to components.json:
{
"registries": {
"@agentsandhumans": "https://agentsandhumans.ai/api/registry/{name}.json"
}
}
Then install eval-comparison (npx shadcn@latest add @agentsandhumans/eval-comparison) and wire it into the flow where your agent answering "better or worse than what, and for whom" after a prompt change.
Swap snapshots for answers and the loop transfers to any agent that produces output a human reviews. The baseline moves only when a reviewer accepts a change, so no answer is approved twice and the build blocks until the review happens.
A score you cannot open is a score you cannot trust. The judge's reasoning, and the cases where humans overruled it, belong on the screen beside the number.
Three words on the release check beat a dashboard that goes unopened. The open question is who decides: failing the job on a regression is a setting, and a person still owns the call.
A rubric per example hides a cost: a standard that shifts with each case is hard for a team to share or defend. The pass and fail examples beside each rule are what let a team hold the rubric steady across cases.
The single number: an overall score is easy to read and hides the trade a change made. Quality up and safety down do not sum to "better".
Trusting the judge: a model scores the answers, and its reasoning must be in view or the score is a black box. The judge's prompt is part of the product.
Noise or regression: outputs vary run to run, so a small drop on one case may be nothing. Severity ranking needs repeat runs behind it.
Baseline drift: if the baseline is whatever ran last, the comparison moves under the person's feet. The baseline is the accepted answer per case, and it moves only when a person accepts a change.
The Story So Far
Sep 11, 2026Added to the Watching list from the prompt-evals brief and the 2026-09-11 vendor pass; the Chromatic review loop shapes the comparison and release entries.