No Bank API? AI Agents Just Take Control of Your Phone Instead

When Apps Never Open Their Doors, Where Do Agents Begin?

Most banking apps offer no external interfaces — and that's unlikely to ever change. For AI agents, this is a hard wall: any agent that could otherwise handle your transactions gets stopped dead at the login screen. No API means no legitimate programmatic entry point.

Since that door will never open, the alternative becomes interesting — instead of knocking, you hand the phone itself to the agent. That's exactly what a new technical approach does: connect a phone to a Mac via cable, and let the agent drive a real device the same way a human uses a touchscreen.

Turning the Screen into Text, Not Images

The most clever insight in this approach is that it passes screen content to the agent as text rather than images. That distinction may seem small, but its impact is significant.

When the screen is transmitted as an image, the agent has to rely on a vision model to "guess" what's in the pixels — which element is a button, which is an input field — with inevitable recognition errors. When the screen is presented as text, the agent reads real labels: it knows what each element is called and what it does, rather than making probabilistic inferences about a visual scene.

Even more interesting: for apps that block screenshots, this approach can still render the interface as wireframes rather than actual screen captures. That means sensitive apps that block image capture for security reasons can still be structurally "read."

The technical foundation of this "text-based reading" is the system-level Accessibility Tree. To support visually impaired users with screen readers, both iOS and Android maintain a tree structure behind every UI element that describes its semantics — recording button names, input field types, element hierarchy, and other metadata. Through a USB connection and debugging protocols (such as iOS's XCTest framework or Android's UIAutomator), external programs can read this tree directly without any image recognition of screen pixels. This is architecturally distinct from the "screenshot → visual understanding → action" approach used by GPT-4V or Gemini: the former is deterministic structured reading, the latter is probabilistic perceptual inference. This is why, even when an app disables the screenshot API (common in payment and banking apps), the accessibility tree remains readable — and wireframe generation is essentially a visual reconstruction of that tree, not a capture of screen pixels.

The Honest Third Outcome: Unknown

Every tap action returns one of three clearly labeled states: success, failure, or a genuine unknown.

That third answer is the most honest one. Many automation tools deliberately avoid the "uncertain" state, preferring to give a definitive success or failure judgment even when they don't actually know the outcome. This kind of false certainty is especially dangerous in high-stakes scenarios like financial transactions — whether a transfer went through is not something you can afford to be vague about.

Honestly admitting "I don't know the result of this tap" actually makes the whole system more trustworthy. It returns judgment to the user rather than masking risk behind artificial certainty.

Replayable, Open Source, and Free

The tool also supports replaying recorded operation flows — meaning a workflow only needs to succeed once, and it can be replayed afterward without repeating the trial-and-error cost. For tasks that need to run periodically, this is a practical efficiency gain.

It's licensed under the MIT License, completely free, and communicates with agents through a tool server with a relatively standardized integration interface.

The tool server mentioned here corresponds to MCP (Model Context Protocol) or similar tool-calling protocols in the AI agent ecosystem. The workflow is: an agent (like Claude or GPT) sends instructions to the tool server via a standardized function-calling interface (e.g., "tap at coordinates X,Y" or "type text into the input field"), the server translates those instructions into actual operations on the physical device, and returns the execution result (including the success/failure/unknown states described above) back to the agent. The advantage of this architecture is that the agent itself is decoupled from the device control logic — swapping out the agent model or device type doesn't require rewriting the core logic. The replay feature is also implemented at the tool server layer: by recording operation sequences as JSON logs, it can re-execute them without invoking the large model again, saving inference costs.

Where the Real Costs Lie

There's no free lunch. The tradeoffs here are concrete: you need a Mac, a cable, and approximately four seconds per tap.

Four seconds per tap means this isn't suited for high-speed batch operations. It's more targeted at constrained environments where there's no other option — when the target app is firmly closed to any interface, physically controlling the device becomes the only viable path.

A Direction Worth Thinking About

This approach reveals an underappreciated reality in deploying AI agents: not all systems are willing or able to provide APIs. Banks, government services, and various closed ecosystems may remain locked down indefinitely.

Rather than waiting for these systems to open up, the angle shifts — let agents use devices the way humans do. Text-based interface reading, honest state feedback, and replayable execution logs turn "the phone as the interface" from a rough idea into an architecturally sound engineering solution.

As the original author put it: your bank was never going to open the door for you. Now, the phone itself is the door.