← Field Notes
OCT 30 · Paper · via arXiv ContextMemoryPermissions

Attackers can rewrite agent memory through ordinary conversation

Memory UI has treated provenance as optional. This research shows that omitting it gives a social attacker the same write access to a human's preference record as the human has. That changes what the memory pattern must guarantee.

Machine summary of the source

Security researchers found and tested a new way to corrupt what AI agents remember about their users. The attack works through conversation. A bad actor poses as a trusted source and convinces the agent to overwrite or invent preference records about a target human. The researchers tested this across five memory systems and six AI models. The attack succeeded at meaningful rates across all of them. The core problem is simple. These memory systems record what an agent learns about a human, but they do not record who told them that. A preference that came from the human looks identical to one a third party inserted. The agent cannot tell the difference. It cannot flag the problem or ask the human to confirm before updating a record. This is a direct challenge to any agent product that stores preferences. The researchers argue that each memory entry needs a source note showing which party supplied it and under what conditions. Without that, the human cannot inspect what the agent believes, cannot correct a poisoned record, and cannot know their preferences were rewritten.

The summary above is generated; the note at the top is the editorial judgment. Primary source ↗