The Future of Work — an eval experience for agentic design workflows

A Chromatic-style review gate for agents, staffed by agents and humans. Part of agentsandhumans.ai. The evaluators assigned to a scenario are called the bench.

1. Why this exists

Chromatic solved a specific problem for UI teams: every commit renders every story, the render is diffed against an approved baseline, a reviewer accepts or denies each change, and the pull request cannot merge until the review is complete. The review is the gate, the baseline is the contract, and the diff makes the change legible to someone who did not write the code.

Agentic experiences in enterprise design workflows have the same shape of problem and none of the tooling. A design-ops agent that migrates tokens, audits accessibility, localises a screen, or turns a Figma frame into a component produces output that is:

• non-deterministic — the same task on the same input can produce a different trajectory;
• multi-modal — a transcript, a set of tool calls, and one or more rendered artifacts (a frame, a component, a doc, a PR);
• consequential — the agent can write to a source of truth (a Figma library, a repo, a CMS);
• opaque to the reviewer — nobody reads a 40-step trajectory unless something is highlighted.

The Future of Work treats an agent run like a story snapshot. A scenario is rendered under every relevant condition, the resulting run is captured, compared against an approved baseline, and the drift is reviewed by a bench of evaluators — some of them agents, some of them humans — until a verdict is reached. The verdict gates whether the agent (or the prompt, or the model, or the tool policy) ships.

What "like Chromatic" means, concretely

2. Who it is for

3. Core concepts

Scenario

A scripted task with a fixture and a success definition. Example:

scenario: localize-de
title:    Produce the de-DE variant of Order Summary
fixture:  figma://Checkout v3 / Order Summary / Desktop (node 12:884)
          [email protected], copy-deck@2026-09
task:     Duplicate the frame, translate copy, resolve text expansion
          within the layout rules, hand back a review link.
success:  - Source frame untouched (write-scope policy WS-2)
          - All strings from the approved copy deck
          - No truncation; layout rule L-4 respected
          - Contrast ≥ 4.5:1 on every text node

Condition

One point in the variant space. The Future of Work declares axes per suite; the product of chosen values is the matrix. See §4 for the full taxonomy.

Run

The captured evidence for one scenario × condition:

• Trajectory — ordered tool calls with arguments, results, and reasoning summaries.
• Artifacts — rendered outputs. Figma frames and components are rendered to PNG + a semantic tree; code is rendered via Storybook/Playwright; documents are rendered to HTML.
• Telemetry — tokens in/out, wall-clock latency, cost, retries, tool errors.
• Provenance — model ID, prompt hash, tool policy hash, fixture hash, seed, git SHA.

Baseline

The most recently accepted run for that scenario × condition. Baselines are per-branch, inherit from the base branch (as in Chromatic), and can be pinned.

Drift

The diff between a run and its baseline, computed on three layers:

1. Trajectory drift — added / removed / reordered tool calls; changed arguments; changed write targets. Classified by severity: a new write is always at least "review", a write outside the declared scope is "critical".
2. Artifact drift — pixel diff (with anti-aliasing tolerance) and semantic diff (token usage, layer names, text content, component instances, ARIA tree).
3. Telemetry drift — cost, latency, tokens vs baseline with per-suite thresholds.

The bench

The set of evaluators assigned to a scenario. Each evaluator is one of:

• Rule grader (agent) — deterministic checks: write-scope policy, token allow-list, contrast, string source, layout rules. Cheap, fast, runs on every run.
• Rubric judge (agent) — an LLM grader scoring against a written rubric with evidence quotes. Runs on every run with drift.
• Human reviewer — pulled in by routing rules (domain touched, severity, sampling rate).
• Human owner — required for verdict on critical drift; can override an agent verdict with a recorded reason.

Verdict and gate

Each evaluator casts Accept / Deny / Discuss. The suite's verdict policy turns individual verdicts into a gate state:

gate = BLOCKED   if any rule grader is critical and no owner override
gate = BLOCKED   if any required human has not voted
gate = BLOCKED   if any human denies
gate = REVIEW    if any rubric judge < threshold and no human accept
gate = PASSED    otherwise

Accepting a run promotes it to baseline for that scenario × condition on that branch.

4. Variant taxonomy

The point of the matrix is to make non-determinism legible. Every axis below is one The Future of Work should be able to declare; each suite picks the subset that matters. Cost grows multiplicatively, so the taxonomy also records what each axis costs and how to prune it.

Matrix explosion guard. A suite of 6 scenarios × 16 axes at 2–3 values each is millions of runs. The Future of Work requires each suite to declare a primary slice (the cells that run on every change), scheduled slices (nightly/weekly), and on-demand slices. Selective re-run (§7) further prunes by input hash.

5. Risk and concern register

Each risk names who is best placed to catch it. "Agent" means a rule grader or rubric judge; "Human" means it needs judgement, accountability, or context that is not in the run; "Both" means an agent detects and a human decides.

5.1 Consequential actions

5.2 Output quality and fidelity

5.3 Evaluation integrity

5.4 Data, privacy, and governance

5.5 Cost, latency, and operations

5.6 Human factors

6. The experience

6.1 Surfaces

1. Suite overview — every scenario × condition as a matrix. Cells encode state: unchanged, drift (needs review), accepted, denied, error, flaky, new. The matrix is the Chromatic "build" page.
2. Run review — three panes: - Trajectory — baseline vs candidate, step by step, with added/removed/changed markers and write targets highlighted. Agent reasoning is collapsed by default. - Artifact — rendered before/after with pixel diff overlay and a semantic diff list (tokens, components, strings, contrast). - The bench — every evaluator, their verdict, their evidence; the viewer's own verdict controls; the gate state and what would change it.
3. Risk register — the living list above, filterable by category, severity, and detector; each risk links to the graders and routing rules that address it.
4. Baseline history — who accepted what, when, with which policy; one-click revert.
5. Suite settings — axes, slices, verdict policy, routing rules, rubrics, retention.

6.2 Interaction principles

• Evidence before narrative. The agent's own summary of what it did is never the first thing a reviewer sees.
• Severity sets the order. Critical drift at the top, unchanged cells collapsed.
• Agent verdicts are labelled and second-class. They can block; they cannot pass the gate alone.
• Every verdict carries a reason. Deny and Discuss require text; Accept on critical drift requires an owner.
• The gate is honest. Partial matrix runs say so; skipped slices are listed.
• Reproduce in one click. Every run has a command that re-runs it locally with the same provenance.

6.3 Where agents and humans sit

7. Architecture

┌─────────────┐   ┌─────────────┐   ┌──────────────┐   ┌──────────────┐
│  Trigger    │──▶│  Runner     │──▶│  Capture     │──▶│  Renderer    │
│ PR / cron / │   │ scenario ×  │   │ trajectory,  │   │ Figma → PNG  │
│ manual      │   │ condition   │   │ artifacts,   │   │ + semantic   │
└─────────────┘   │ in forked   │   │ telemetry,   │   │ tree; code → │
                  │ fixtures    │   │ provenance   │   │ Storybook;   │
                  └─────────────┘   └──────────────┘   │ docs → HTML  │
                                                       └──────┬───────┘
┌─────────────┐   ┌─────────────┐   ┌──────────────┐          │
│  Gate       │◀──│  Bench      │◀──│  Differ      │◀─────────┘
│ PR check,   │   │ rule graders│   │ trajectory / │
│ deploy hook,│   │ rubric judge│   │ artifact /   │
│ baseline    │   │ human route │   │ telemetry    │
│ promotion   │   │ verdict log │   │ drift + rank │
└─────────────┘   └─────────────┘   └──────────────┘

• Runner executes the agent under test inside an isolated fixture: a forked Figma file, a throwaway branch, a sandboxed CMS. Tool policies are enforced at the tool-proxy layer, not by asking the agent nicely.
• Capture records every tool call and result as structured JSON (open schema), with hashes of every input that could change behaviour.
• Renderer produces a viewable artifact for each output type. Figma outputs render via the REST image API plus a node-tree export; code renders through the existing Storybook + Playwright pipeline (this is where Chromatic itself can plug in); documents render to HTML.
• Differ computes drift on the three layers and ranks it.
• Bench runs rule graders, then rubric judges (different model family from the agent under test), then applies routing rules to request human review. Verdicts land in an immutable log.
• Gate exposes a status check (GitHub, GitLab), a webhook for deploy pipelines, and baseline promotion.
• Selective re-run hashes the inputs of each scenario × condition (prompt, tool policy, fixture, design-system version, model) and re-runs only cells whose hash changed.

Integrations for enterprise design workflows

Figma (REST + plugin for fixture forking), Storybook/Chromatic (code artifacts), GitHub/GitLab (status checks, PR comments), Jira/Linear (Discuss verdicts open tickets), Slack (routing notifications), SSO/SCIM, SIEM export of the verdict log.

8. Rubrics

Rubric judges score against written rubrics with mandatory evidence quotes. A starting set for design-ops agents:

• Scope discipline — did the agent do what was asked and nothing else? Writes only to declared targets; no side-effects.
• System fidelity — tokens, components, and patterns from the design system; no hard-coded values; correct variants.
• Layout integrity — auto-layout preserved; no truncation; responsive rules respected.
• Content fidelity — strings from the approved source; no invented content; correct locale rules.
• Accessibility — contrast ≥ 4.5:1 (3:1 for large text), focus order, labels, target sizes.
• Communication — the agent's hand-back is accurate; it flagged what it could not do; it did not claim completion it did not achieve.
• Efficiency — within token and latency budgets; no redundant tool calls.

Scores are 1–5 with a per-item threshold; "Communication" is scored by comparing the agent's self-report against the captured trajectory, which is the single best detector of fabricated completion.

9. Metrics

10. Roadmap

Phase 0 — Prototype (this repository). A static, interactive prototype of the run review and suite matrix with realistic sample data, living at agentsandhumans.ai. Purpose: align designers, engineers, and reviewers on the shape of the experience and collect the risk register.

Phase 1 — Capture and diff. Runner + capture in isolated Figma and repo fixtures for three scenarios; trajectory and telemetry drift; rule graders for write-scope, tokens, contrast, strings. Verdicts are human-only. Output is a GitHub status check.

Phase 2 — The bench. Rubric judges on a different model family; routing rules; the three-pane review; baseline history; Discuss → ticket.

Phase 3 — Matrix and pruning. Declared axes and slices; selective re-run; flake scoring; fault injection; scheduled slices.

Phase 4 — Governance. SSO, audit export, retention, data-residency per suite, red-team scenarios, human-coverage reporting.

11. Open questions

1. Who owns the baseline when a scenario spans design and engineering? Proposed: the scenario declares an owner role; the routing rule names the person.
2. How much of the agent's reasoning to capture. Full reasoning is useful for debugging and a liability for retention. Proposed: capture summaries by default, full reasoning on critical drift only, with per-suite retention.
3. Should agent verdicts ever pass a gate alone? Proposed: only for unchanged cells and only with a human sampling floor.
4. Pixel diff tolerance for generated design artifacts. Font rendering and anti-aliasing differ between Figma exports; semantic diff should lead and pixel diff should support.
5. Rubric drift. Rubrics need versioning and their own review gate.
6. Adversarial fixtures. Who maintains the prompt-injection corpus for design files, and how is it kept out of the agent's training data?

12. Glossary

Scenario a scripted task and fixture · Condition one point in the variant space · Run the captured evidence for a scenario × condition · Baseline the last accepted run · Drift the diff between a run and its baseline · Bench the evaluators assigned to a scenario · Verdict Accept / Deny / Discuss with a reason · Gate the status that blocks or allows a change · Slice a named subset of the matrix · Rule grader a deterministic agent check · Rubric judge an LLM grader with a written rubric · Owner the human who can override and promote.