← Field Notes
NOV 16 · Clipped · via Vercel Blog Production DriftRelease DecisionEval Set Authoring

Vercel argues agents need their own software testing track

The human's judgment used to live in their head. This model asks them to write it down in a form a test can use. That gap between 'I know good when I see it' and a runnable standard is the new design problem.

FIG. 1 · Where this sits in Production Drift
    4 NOTES ON PRODUCTION DRIFT IN 26 WEEKS · SCRUB THE WEEKS, CLICK ONE TO PIN IT
    Published, annotated field notes attached to Production Drift, by week observed · last 26 weeks · n = 4. Rail markers are lifecycle events from the pattern changelog.
    Data table
    Week ofNotes
    MAY 25Langfuse puts the verdict on the release check in three words
    SEP 14An RCT caught a deployed chat agent breaking its own written rules
    SEP 21Comcast ran 140 million simulated calls before shipping its CX agent
    NOV 16Vercel argues agents need their own software testing track

    Machine summary of the source

    Vercel published a case for treating agents as a distinct class of software user. An agent calling your service behaves in a different way from a human clicking through it, so a separate testing track is needed to know whether your service works for agents. That reframe has direct consequences for how teams build and validate software that agents will use. The practical shift is in what you test and when. A human can recover from an ambiguous response or a broken flow. An agent often cannot. It may misread a response, take a wrong branch, and compound the error across many subsequent steps. Vercel's position is that you need test cases written around agent behavior, run against a defined configuration, and compared over time. That lets you see quality drift before it reaches production. For product and design teams, the people building agent-facing surfaces need to write down what good looks like, in the form of concrete examples a test can check. That work, turning a designer's or product manager's judgment into a repeatable standard, is the new craft this model demands. Without it, a team has no reliable signal that a prompt change helped or hurt the agents depending on it.

    The summary above is generated; the note at the top is the editorial judgment. Primary source ↗