← Field Notes
OCT 13 · Clipped · via GitHub Emergency StopPermissionsReach Preview

A $7 ESP32 chip lets OpenAI vision models tap and swipe an iPhone

The phone's owner grants access once by plugging in the board, then has no view of what runs. That is the sharpest version of the reach and approval problem I have seen at the hardware layer.

Machine summary of the source

A hobbyist project called esp32-iphone-agent uses a $7 microcontroller board to give any OpenAI-compatible vision model direct touch control over an iPhone. No app install. No iOS configuration profile. The model sees the screen, decides what to tap, and the chip sends the signal. The phone cannot tell software from a finger. This breaks a core assumption in mobile operating system design: that the permission model protects the device from unsanctioned control. The OS grants or refuses access to apps. A piece of hardware that mimics a finger sidesteps that model. The human who sets up the rig decides what the agent can reach, but nothing on the phone can stop it or log it afterward. For teams building agent products, the gap is between what a human intends to grant and what the agent can do. The human sets up the hardware, then walks away. From that point, the agent acts with no record the phone can inspect, no confirmation step, and no way for the phone's owner to see what ran. The design work is in building the layer where a human can see what the agent did, narrow its reach before it starts, and stop it if the work goes wrong.

The summary above is generated; the note at the top is the editorial judgment. Primary source ↗