Researchers at Stanford built a benchmark to test whether AI agents could resist social engineering, the kind of manipulation a scammer uses to get a person to hand over credentials or take a harmful action. The benchmark placed agents in realistic scenarios where an attacker tried to deceive them into doing something the human operator would not sanction. Most agents, including leading large language model-based systems, failed a meaningful share of these tests, taking harmful actions when a plausible-sounding but malicious instruction appeared in their environment.
The findings matter for product teams because an agent that acts on instructions from its environment, not just from its designated operator, is an agent that can be hijacked. The research shows this is not a theoretical risk. Agents that lack a clear model of who is allowed to instruct them, and what kinds of requests should trigger a pause or refusal, become a liability the moment they touch anything external, an email inbox, a web page, a document, a calendar invite.
The design implication is direct. Agents need a legible model of trust: which instructions come from the operator, which come from the environment, and which should never be followed without a human confirming. Building that into an agent product means giving the human a way to see and set those boundaries before the agent runs, and a fast path to stop it when something goes wrong.