← Field Notes
OCT 13 · Paper · via Martin Fowler Budget ScopingRelease DecisionEval Run SetupReview

Fowler and Jain measure what scaffolding costs to run an AI coder

The harness is the signal the reviewer depends on. If it's too expensive to run or too noisy to trust, the human's review step loses its foundation before it begins.

Machine summary of the source

Martin Fowler and Rouan Jain ran a controlled experiment to find out whether the test harness (the automated checks that verify an AI coding agent's output) is worth building and running. Their answer: yes, but the cost and reliability of that harness decide whether the whole system is viable. A harness with weak design can cost more to run than the agent work it validates. A flaky harness produces false signals that erode trust in the whole system. The experiment produced concrete numbers. Running a full harness on each agent step was expensive. Running a lighter check more often and a heavier check less often cut costs without sacrificing confidence. The authors found that harness design, meaning what to check, how often, and at what fidelity, is a first-class engineering decision. A team that treats harness design as secondary ends up either overspending or flying blind. For a product team building with AI agents, the lesson is direct. The checks you build around an agent shape how much you can trust its output and how much that trust costs you. Designing those checks is design work. The humans who review agent output need the harness to give them a clear, fast signal. A noisy or expensive harness puts that review in jeopardy before the human ever sees the result.

The summary above is generated; the note at the top is the editorial judgment. Primary source ↗