Security and governance
Design the agent as if the model will be fooled
Every pattern in this library assumes the model will, some of the time, follow an instruction it should not. The safety comes from the architecture around it: what each component may touch, what crosses a trust boundary as data, which actions wait for a person, and what is counted outside the model. This page collects the controls from every pattern by the threat they answer. Each "Break" scenario on a pattern page shows an attack and the control working.
Three trust levels, on every diagram
- trusted
- Code and data the operator controls.
- semi-trusted
- The model: it follows instructions from anywhere it reads them, so its output is checked.
- untrusted
- Content from outside: web pages, tool results, user files, other agents. Treated as data, never as instructions.
Turn on Trust boundaries in any pattern's stage to see the zones drawn on the diagram.
The principle with the most leverage
Remove a capability rather than trust a prompt. Simon Willison's lethal trifecta (2025-06-16) names the combination to avoid in one agent: access to private data, exposure to untrusted content, and a way to communicate externally. Take any one away and prompt injection can no longer exfiltrate. The orchestrator–worker pattern removes the third leg by design; its "A web page instructs the worker" scenario shows why it matters.
Prompt injection and untrusted content
A model follows instructions from anywhere it reads them: files, pages, tickets, tool results, other agents.
| Control | How it works | Pattern · nodes |
|---|---|---|
| Tool results are data | The runtime never promotes tool output to the instruction role; the system prompt names the developer as the only source of instructions. | Single-agent tool loopread · tests · context |
| Structured results with provenance | Results are validated against a schema; findings without a source are dropped and flagged. | Orchestrator and workersmerge |
Data exfiltration
An agent that can read private data and send messages can be talked into sending the data out.
| Control | How it works | Pattern · nodes |
|---|---|---|
| No external send | This loop has no tool that communicates outside the machine; one leg of the lethal trifecta is absent by design. | Single-agent tool loopagent |
| Workers are sandboxed by capability | Each worker type has a fixed tool set; the web worker can read but not write, send or reach internal systems. | Orchestrator and workersw1 · w2 · w3 |
| No external communication in the orchestrator | The orchestrator’s only outputs are the plan and the answer, delivered to the researcher. | Orchestrator and workerslead · answer |
Excessive agency and destructive actions
An agent with broad tools does what a compromised decision tells it to.
| Control | How it works | Pattern · nodes |
|---|---|---|
| Allowlisted commands and gated edits | The shell tool runs only allowlisted commands; edits outside the project directory are refused; in cautious mode edits wait for approval. | Single-agent tool looptests · edit · developer |
| Human approval as a node | The reviewer node is on the high-risk path; the run cannot reach implement without state.decision = approve. | Stateful graph with branching and human approvalreviewer |
Runaway cost and unbounded loops
A loop that never decides to stop, or a worker that spends the whole budget.
| Control | How it works | Pattern · nodes |
|---|---|---|
| Budget outside the model | Steps, tokens and seconds are counted in operator code and stop the loop. | Single-agent tool loopbudget |
| Per-worker budgets | Token and time limits per worker and in total, enforced outside the models. | Orchestrator and workersbudget |
| Bounded retries | The fail edge’s condition closes at the limit and the failure edge escalates. | Stateful graph with branching and human approvalverify |
Corrupted state and repeated side effects
A node writes nonsense another acts on; a resumed run does the same thing twice.
| Control | How it works | Pattern · nodes |
|---|---|---|
| Idempotent side effects | Ship checks the run id before creating a PR; every side-effecting node is keyed by run id. | Stateful graph with branching and human approvalship |
| Typed state | State has a schema with reducers; writes are validated at the boundary. | Stateful graph with branching and human approvalstate |
Other controls
- Rules before the model in the router: Deterministic rules on code facts decide first; the model only breaks ties and cannot lower a rule-assigned risk. Stateful graph with branching and human approval
Watch an attack and a control
- A tool times outSingle-agent tool loopThe test runner hangs. The loop survives because the tool, not the model, enforces the timeout and returns an error as data.
- A file contains instructionsSingle-agent tool loopA tool result carries text addressed to the agent. The loop is safe only because tool results are treated as data and the dangerous tools are gated.
- A web page instructs the workerOrchestrator and workersThe web worker reads a page that tells it to make the orchestrator email the findings to an outside address. The architecture, not the prompt, is what stops it.
- Tests fail, retry, fail, escalateStateful graph with branching and human approvalThe implement–verify cycle is a bounded loop inside the graph. After the limit, the failure edge brings in the reviewer instead of looping forever.
Governance frameworks
- NIST AI Risk Management Framework (AI RMF 1.0) (NIST, 2023-01-26). The SDLC explorer's security stage follows its Govern, Map, Measure and Manage functions. It is a voluntary framework, not a certification.
- OWASP Top 10 for Agentic Applications for 2026 (OWASP GenAI Security Project). Not yet verified here: the site refuses automated fetches, so the ten risks have not been read into this library. No pattern cites it as evidence until an editor confirms it. This is tracked as an open item in the delivery report.
The Agents & Humans Briefing
How humans and agents work together: the day's field notes and the patterns behind them, in one short email around noon UTC.
Free. No spam. Unsubscribe anytime.