← Field Notes
NOV 18 · Paper · via arXiv ApprovalConfidence SignalingHuman Handoff

Shopping agents sound more certain but pick no better products

The agent gets louder as it gets more wrong. Confidence display becomes a liability when it tracks effort, not accuracy. Approval needs the agent's reasoning in view, not its certainty score.

FIG. 1 · Where this sits in Approval
    65 NOTES ON APPROVAL IN 26 WEEKS · SCRUB THE WEEKS, CLICK ONE TO PIN IT
    Published, annotated field notes attached to Approval, by week observed · last 26 weeks · n = 65. Rail markers are lifecycle events from the pattern changelog.
    Data table
    Week ofNotes
    AUG 24Model Context Protocol servers get a way to hand off to the operator · 150 participants spotted the dark patterns and complied anyway · Margaret Mitchell argues oversight wears down as agents do more alone · 113 participants wrote their own agent rules and got overreach anyway · A Cloudflare engineer gives agents code to run instead of a tool list · Simon Willison shows agents turning a bug rumour into a live exploit · Claude broke into three companies' networks and posted malicious code · Johann Rehberger got past Claude Code's auto mode in 80% of runs
    AUG 31OpenAI's agents broke out and hijacked a German website on their own · The Model Context Protocol drafts requests that pause for the operator
    SEP 7A poisoned tool description steered agents across 25 test setups · Seven agents ran real businesses and sent $12,431 in unasked invoices · Repeat-After-Me hides orders in images and rewrites an agent's tools
    SEP 14Gemini broke out of its task and hacked three companies · Suganthan watched Claude pay a penny to read his website · Blindspot tests whether AI agents stop too soon or run too far · arXiv paper finds AI agents leak secrets when they police themselves · AcquireBound makes agents ask before grabbing new access · 89 sources later, researchers lay out who should approve what
    SEP 21Meta's new agent looks safe because it asks before it acts · 1,248 GitHub agent specs show safety steps missing from most · Anthropic's coding agent signed a contract without developer approval
    SEP 28NarrativeSteward lets writers hand off and reclaim story work from AI · Consumer AI agents fail most tasks when left to run alone
    OCT 5Vercel lets agents buy domains but makes them stop for your yes · Anthropic's Claude filed a false murder tip with Philadelphia police · arXiv researchers show how poisoned data hijacks multi-agent work · Monitors missed harmful AI actions 63% of the time in new study · ArXiv paper finds agents act on plan steps you did not see · Attackers fool AI code reviewers by rewriting the attack mid-review · 18 coders explain when they let an agent act and when they don't · GitHub Copilot learns to click and type in desktop apps
    OCT 12EAIR paper finds enterprise agent teams break rules together · Hidden ads in search results steered what a shopping AI bought · Amex lets cardholders cap what an AI agent can spend before it shops · Researchers show how shared agent skills carry poison across sessions
    OCT 19Vercel gives its agent narrow, revocable access to production · DoorDash ran 35,000 code changes through AI agents for $4.79 each · Open index maps how much of a job agents can take over · HCI researchers map how shared workspaces change agent work · Model strength sets who leads, not screen design · Gaze tracking reveals how humans miss agent mistakes in real time · One true fact in a forged message fools 18 AI agents into compliance · Cling beats full-agent automation by letting humans steer each step · ArXiv researchers use web structure to stop agents being tricked · GatewayBench puts agent routing servers through adversarial testing · Anthropic's AI ran a biology lab's image analysis start to finish
    OCT 263,000 turns show coding agents rewrite far more than you asked · FlashScript turns repeated agent runs into fixed, reusable programs · 31 orgs confirm code review breaks when agents write the code
    NOV 2Apiiro found 1,122 GitHub files holding passwords after one approval · Researchers argue shared agent parts make each step easier to review · OpenAI patched a thousand Linux vulnerabilities without human sign-off · One AI helper can deceive another before you see the result · PrivacyFlow maps where agents leak personal data before damage is done · Interface Quilt adds buttons to live apps without touching their code
    NOV 9Adversaries learned to weaponize an agent's refusal to act · ShadowBench finds agents calling services operators did not approve · Stanford tests find AI agents duped by realistic scam instructions · Simon Willison had Claude write code to find a mystery beep · 289 workers sorted tasks they would delegate from tasks they kept
    NOV 16Study finds reviewer expertise matters more than reviewer presence · Shopping agents sound more certain but pick no better products · Researchers find AI agents can be redirected to a fake tool by name · 108 crisis counselors showed where AI drafts help and where they hurt

    Machine summary of the source

    Researchers tested shopping agents built on large language models. They found that asking an agent to reason longer and harder raised its expressed confidence. It did not raise the quality of its purchase decisions.

    This creates a specific problem for designers. The agent sounds more certain, so the human is more likely to approve. The underlying answer is no better than a faster, cheaper one would have been. The mismatch between confidence and correctness is the design risk.

    The implication for teams building shopping or procurement agents is direct. Surface the agent's confidence score with care. High confidence after extended reasoning is not evidence of a better answer. Designers should consider showing the human what the agent considered, not how hard it thought, so the human can judge the choice on its merits.

    The summary above is generated; the note at the top is the editorial judgment. Primary source ↗