$7 ESP32 Lets AI Control iPhone: A Local Vision Model in Practice

Apple spent 17 years convincing developers that controlling an iPhone requires Xcode, a $99 developer account, and a desk more expensive than your couch. Henry Yuan proved with $7 that none of that holds.

His open-source project esp32-iphone-agent appeared on the timeline on October 11 as a 12-star "shitpost" — "a $7 ESP32 can let an AI agent control a non-jailbroken iPhone." Hackaday followed up on October 14. By the time I cloned it, it had 2,975 stars. You're not too late yet.

The essence of this project isn't the AI — it's the hardware-level "heresy." Yuan breaks the classic computer-use agent into three organs, each assembled from off-the-shelf parts:

• Eyes: AirPlay mirroring
• Hands: an ESP32-S3 disguised as a USB mouse and keyboard
• Brain: any vision model compatible with the OpenAI API

No WebDriverAgent, no CoreDevice exploits, and no on-device app requiring Apple's signature. Just a microcontroller that iOS inherently trusts — because to the system, it's a mouse.

I rebuilt this system for the realities of an independent creator: instead of two DGX Sparks running Qwen3-VL-Flash as in Henry's demo, I used a consumer GPU with Ollama and a 7B vision model. The README glosses over the scariest part, so I'll focus on the security model — because what we just built is essentially a Wi-Fi-controlled USB HID injector.

Eyes: UxPlay Plus a GStreamer JPEG Pipeline

The original v0.1.0 architecture — which most people will still build because it's easier to debug — has a wasteland kind of beauty.

iPhone → (Wi-Fi) AirPlay → Mac/Linux host running UxPlay → GStreamer → JPEG → your agent.

Yuan's Hackaday description is blunt: screen mirroring is captured by UxPlay, then "a GStreamer pipeline feeds the screen as a JPEG stream to any AI agent." UxPlay is an underrated hero: it's a real AirPlay receiver built on GStreamer plugins, so your iPhone thinks it's just casting to a TV.

In the new firmware, Yuan gets wilder: the ESP32-S3 opens its own hotspot called ESP32 Agent, acts as the AirPlay receiver itself, and sends H.264 over BLE to an iPhone app. The phone keeps cellular data for the model while Wi-Fi is "hijacked."

Both approaches give you what Xcode automation can't: a real framebuffer. Not a screenshot API — the actual rendered screen, at roughly 5fps, 480x1040. Low enough for BLE to handle, high enough for a vision-language model to read. Feed those base64 images to /api/v1/chat/completions and that's your eyes.

The AirPlay mirroring protocol is Apple's proprietary screen-casting standard, based on RTSP negotiation and H.264 video streaming. UxPlay, as an open-source implementation, reverse-engineers the receiver logic so non-Apple devices can pose as an Apple TV and receive the iPhone screen. Its technical core lies in the GStreamer multimedia framework: a pipeline decodes the raw H.264 stream, scales it, encodes it into a JPEG sequence, and feeds it via base64 into the vision model's image input. The value of this chain is that it bypasses iOS's screenshot API limitations — the screenshot API requires the app to be in the foreground and authorized, whereas AirPlay mirroring captures the system-level framebuffer, a lower-level "eye" for automation agents.

Hands: ESP32-S3 as USB HID, and the Critical Role of AssistiveTouch

This is the part that made me laugh out loud.

Apple will never let a random USB device tap the screen. But it will let an accessibility mouse move the cursor. Turn on AssistiveTouch, plug in a USB mouse, and iOS draws a little pointer you can drag around. Tap = tap, drag = swipe.

Yuan's firmware does exactly this: the board is a standard USB mouse and keyboard. With AssistiveTouch enabled, the mouse moves an on-screen pointer, so clicking is tapping and dragging is swiping. I flashed a Seeed XIAO ESP32S3 — the regular version, 8MB PSRAM, not Sense or Plus, about $7. Any ESP32-S3 with native USB and PSRAM works. iOS 18+ connects directly via USB-C; Lightning ports need that cursed $30 camera adapter.

The board exposes two HID interfaces via TinyUSB: a relative mouse and an ASCII keyboard. The computer tells it over Wi-Fi (older version) or BLE (newer version) to execute move x y, click, type "hello", and it obeys.

The genius is pointer drift. Because the mouse uses relative coordinates, every move accumulates error. Yuan's app tracks the error, re-homes in a corner, and wiggles the pointer before precise clicks to find it on screen and correct the offset. That "wiggle" is why my automation stopped missing that tiny x in Safari.

The USB HID (Human Interface Device) protocol is a category in the USB standard defining human-interaction devices: keyboards, mice, and gamepads all belong to it. Operating systems inherently trust HID devices: plug and play, no driver signing or user authorization required. The ESP32-S3 emulates HID descriptors via the TinyUSB software stack, making iOS recognize it as a standard mouse and keyboard — this is precisely the physical-layer breakthrough that bypasses Apple's app-signing system. The attack surface is that the HID protocol itself contains no device authentication mechanism; any microcontroller that can enumerate as HID can inject input events.

Swapping the DGX Spark for Ollama on a Consumer GPU

The README is honest: "The project was developed using nvidia/Qwen3.8-Flash-Next-NVFP4, deployed on two DGX Sparks (TensorFold)." That's cool if you have $20k to burn.

My creator setup:

• Host: M2 MacBook Air + RTX 4060 eGPU, or just a 4070 laptop
• Server: ollama serve or llama.cpp --server with vision
• Model: qwen2.5vl:7b or minicpm-v:2.6. Qwen is more accurate at spatial grounding; MiniCPM is faster.
• API: any OpenAI-compatible chat API that accepts images — a cloud API or your own server

In the iPhone app, you can set separate planner and executor models — and on a single GPU you absolutely should. I used Qwen2.5VL 7B as the executor (fast clicks) and, over Wi-Fi, a cloud-quantized Qwen2-VL 32B as the planner, or locally the same 7B with a longer system prompt.

Per-step latency for the local 7B, from JPEG input to HID command output:

• Eyes (AirPlay→JPEG): 180-250ms
• Visual inference (Ollama 7B Q4_K_M on RTX 4060): 1.8-2.4s
• Hands (Wi-Fi→USB HID→pointer stabilization): 350-600ms
• Screen stabilization wait: 800ms

Total: about 3.5 seconds per step. Cloud GPT-4o takes only 1.2s, but it leaks your banking screen to OpenAI. Local is slower, but it's yours. BLE bandwidth is about 300-500kbps, so screen mirroring is small (480x1040, ≤5fps); each step takes seconds, mostly spent on the model — that's not a bug, it's physics.

Planner and Executor: One Model or Two

Yuan ships two prompts in prompts/ — plain text, bundled in the app, recompile after editing.

This matters more than you think.

• Single-model approach: one VLM receives "Goal: set an alarm for Monday/Tuesday at 8am. Current screen: … what action?" and outputs {"action":"tap", "x":512, "y":1340}. Simple, but after 6 steps it forgets why it's clicking and starts looping.
• Dual-model approach: the planner (larger, slower) says "Step 1: open the Clock app. Step 2: tap Alarm. Step 3: …", and the executor (smaller, faster) executes only Step 2 based on the current screen. Yuan's app runs exactly this: decode → vision model → action — one planner plus one executor: tap, swipe, draw, type, key, home, done.

On a single GPU, run the planner every 3 steps, not every step. Cache the plan. The executor makes fewer errors with a short horizon.

My prompt tweak to fix stuck loops:

"If the screen hasn't changed after the last action, you MISSED. Wiggle the pointer, increase the tap radius by 20px, or switch to a swipe. Never repeat the same coordinates."

Add a change detector: after each action, wait for the screen to stabilize and tell the model whether anything changed. If image_diff < 2%, force a re-plan.

The spatial grounding capability of vision-language models refers to the model's ability to map natural-language instructions to image coordinates, typically via bounding-box prediction or coordinate regression. The Qwen2.5VL series outperforms same-scale models on UI element grounding benchmarks (such as ScreenSpot) because its training data includes large amounts of GUI screenshots and element annotations. But at 480x1040 low-resolution input, the 7B model still tends to offset for icons smaller than 40px — this is the root cause behind the article's emphasis on "pointer wiggle calibration": the model outputs a relative coordinate estimate, and the hardware layer needs closed-loop correction.

Latency and Failure Loops: Calibration, Re-planning, Stuck States

Let's be honest: this rig fails hilariously.

Failure modes I logged across 50 tasks:

1. Calibration drift: after 10 clicks the pointer is off by 30px. Fix: Yuan's app re-homes in a corner every 5 actions and wiggles before precise clicks. Do that. Don't trust absolute coordinates — screen width and height are relative, so pair with stored calibration data.
2. iOS black holes: iOS doesn't mirror certain screens (password entry, Apple Pay, DRM video); mirroring stops when the phone is locked. Your agent goes blind instantly. You need a timeout mechanism that pauses the run and resumes from the same step when the connection returns.
3. Keyboard hell: keyboard input is ASCII only. Want to type Chinese? No. It types via USB keyboard events, so you get only what a US keyboard can produce.
4. Stuck states: the model sees an "Alarm set" toggle and assumes it's done, but never tapped Save. Add a finish action that only the planner can call, and require two consecutive finish votes.

My fix loop: save every run on the phone, replay it step by step with click positions drawn, and export trajectories for analysis. Use it.

Threat Model: This Proves How Easy a Wi-Fi-Controlled HID Injector Is

The README says "only automate devices and accounts you own." It should say this in much bigger type: this is a BadUSB with a radio.

You plug a $7 board into an iPhone, and it:

• is powered by the phone
• emulates a mouse + keyboard (so it can type anywhere, including password fields)
• runs a Wi-Fi hotspot the phone auto-joins, with its own AirPlay receiver stack (GPL-3.0 portions)
• accepts BLE commands to drive HID, with firmware you flash via flash.sh

In older versions, commands came from a Python server over Wi-Fi. In the new version they come from the app over BLE, but the board still controls HID. If that BLE characteristic isn't authenticated with a rolling key — and NimBLE-Arduino doesn't do that by default — anyone within 10 meters can command your phone to start typing.

This ties directly into research on hardware implants. The Seeed XIAO is exactly the size of a drop implant: stuff it in a cable, make it look like an adapter, power it forever, and it's network-isolated except for the hotspot it opens itself. Apple trusts it because it's HID. iOS's AssistiveTouch is an accessibility feature, not a security boundary.

Defenses:

• Edit secrets.h at flash time and enable NimBLE Secure Connections
• Never leave the board plugged in unattended
• iOS: Settings → Accessibility → Touch → AssistiveTouch → Devices → forget it when you're done
• If you're red-teaming: remember AirPlay mirroring doesn't capture passwords/Apple Pay — you can't get the Secure Enclave screens, but everything else is fair game

Yuan did responsible disclosure: the repo has a SECURITY.md and responsible-use notes. But the point stands: Xcode-free iPhone automation also means Xcode-free iPhone exploitation.

BOM for an independent build (what I actually used):

• Seeed XIAO ESP32S3 (~$7)
• Lightning-to-USB camera adapter for older iPhones (~$30)
• Mac or Linux host, arduino-cli + ESP32 core 3.x, UxPlay, python venv
• Ollama or llama.cpp with a vision model — I benchmarked qwen2.5vl:7b

Total cost if you have a GPU: $7. Total time: one afternoon to flash, calibrate, and watch a robot try to set an alarm, fail three times, then succeed.

Apple built a walled garden. We just found the mouse hole.

Background Notes

The BLE GATT security model: in Bluetooth Low Energy's Generic Attribute Profile (GATT), a characteristic's read/write permissions are defined by attribute flags, but link-layer encryption or application-layer authentication is not enforced by default. The NimBLE-Arduino stack defaults to "Just Works" pairing with no MITM protection, so an attacker can man-in-the-middle or forge HID commands. Enabling LE Secure Connections requires explicitly configuring key distribution and bonding flags in firmware; otherwise the HID control characteristic exposed by the ESP32 amounts to a passwordless remote keyboard. This is the technical foundation of the article's threat model.

One-line Summary

A $7 ESP32 poses as USB HID and, via AssistiveTouch, lets a vision model control an iPhone without Xcode — but plants BLE HID injection risks.