Local AI Computer Control: It Can Click for You, But Asks You Four Questions First

Locally Uncensored 3.0.6 adds a feature worth taking seriously: letting local models operate apps you approve with the mouse and keyboard. Screenshots and clicks all stay on your machine and are never uploaded to the cloud. But it's currently in beta, and the author has spent most of the engineering effort on "asking clearly before acting"—which is also the focus of this article.

Off by Default—and Enabling It Isn't a Blank Check

Computer control sits under Agent Permissions in settings and is off by default. Turning it on doesn't mean granting global permission. Afterwards, LU will still stop and ask you at four points:

• Every new conversation: "Allow computer control in this conversation?" Allow / Deny
• First operation on a given app: ask before using
• Every action: ask by default (supervised mode)
• Sensitive actions: always ask, in any mode

Until you click "Allow" on the first card, LU doesn't take screenshots or do anything. That "Allow" only applies to the current conversation; a new conversation asks again, and every conversation after an app restart asks again. No setting can skip this card.

"Sensitive" refers to operations like sending, purchasing, deleting, or modifying permissions—when asked, the reason is highlighted in red. Clicking an unnamed button, link, or menu item also counts as sensitive—because LU can't tell what pressing it will trigger.

Forbidden Zones It Never Touches

Some areas are inaccessible at any level: terminals, password managers, system login and permission dialogs, security settings, the Run dialog, and windows with administrator privileges. LU won't type into password fields, and pasting into password boxes is also rejected. Before double-clicking a program, installer, script, or shortcut, it always asks first.

How to Stop It, and What It Looks Like

Pause is Ctrl+Alt+Space, stop is Ctrl+Alt+Esc; on Mac, replace Ctrl+Alt with Control+Option. In autonomous sessions, moving your mouse also triggers a pause. The pause screen shows what it has done so far and asks whether you want to take over or let it continue; you can also append an instruction for the remaining task. A screen task ends after at most 25 steps or 8 minutes no matter what, and the time you spend answering questions counts toward that.

When working on Windows, the LU main window shrinks into a small window in the top-right corner, and a faint purple glow appears at the screen edges. The small window shows what LU is doing and lists the two shortcuts. Questions appear in the small window with a stop button, and only accept mouse clicks—so the app being operated keeps keyboard focus.

Vision vs. Control List: An Interesting Trade-off

By default it uses the chat model you've already selected; LU won't additionally ask how you want to run it. On Windows, LU reads the list of buttons and fields in the target window: models that can see images get screenshots plus the list, while models that can't see images work from the list alone, with no images. It's precisely the latter case that makes small local text models usable here.

The author ran a comparison of three runs each on two tasks, and all six completed: the vision runs totaled 497 seconds, while the list-only runs totaled 381 seconds. In other words, in this scenario "not looking at images" was actually faster.

As for dedicated screen models, the catalog includes Holo 3.1 9B, marked Beta, with a suggested 16GB of VRAM minimum. In the author's tests on Windows, it completed a simple typing task in only one of three runs, and it can't coexist with a 9B chat model on a 12GB card. The author's exact words: "This isn't a strong result, and I'd rather say so than leave it out"—a candor that's uncommon in similar releases.

Control list (UI Automation) is an accessibility API provided by Windows that lets programs enumerate buttons, text boxes, menu items, and other elements within a window in a structured way, and read their names, types, states, and other properties. Compared with screenshots, control lists don't rely on pixel recognition, so purely text-based models can also understand the interface structure and decide on click targets. However, its coverage is limited by whether an app exposes complete UI Automation information; custom-drawn interfaces may not be parsed correctly. This is also why LU can't provide control lists on Linux and macOS and can only rely on screenshots.

A Real Number

Giving a command by voice and having a local 9B model type a note in Notepad based on the control list: from the end of the voice command to the spoken result, it took 11 to 22 seconds. This was measured before the "ask every new conversation" mechanism went live, so in a real new conversation you'd also need to add the time you spend clicking "Allow."

Platform Differences Are Bigger Than You'd Think

Windows has been verified point by point on a Windows 10 machine; Linux hasn't actually been run in the app yet and requires an X11 session; macOS is integrated but not thoroughly tested. Control lists and autonomous mode exist only on Windows. On Linux and macOS, LU can't say what a click will press, so every click is asked as a sensitive action. On Mac you also first need the Accessibility and Screen Recording system permissions, and the model must have image input capability—because only screenshots are available. No platform has a background mode; it works in the foreground with your real pointer.

Limitations Worth Knowing in Advance

• Small local models misjudge when a task is complete, showing as stopped even though the work is done.
• Text on screen may try to give the model instructions. LU marks content read from the screen as untrusted and asks before sensitive actions; but for important operations, stay in supervised mode.
• The wording LU uses to determine whether a button is sensitive comes from a list, and phrasing the list doesn't recognize won't be identified.
• The eight fixes and ten security fixes added after Windows verification have only been run in automated tests.

Brief Conclusion

Locally Uncensored 3.0.6's computer control lets local models click and type in apps you approve. It's off by default, asks every new conversation, asks before sensitive operations in any mode, never touches terminals or password boxes, and Ctrl+Alt+Esc stops it at any time. Windows is verified, Mac isn't thoroughly tested, and Linux hasn't been run yet. Treat it as a beta that needs you watching over it.

The code is at github.com/PurpleDoubleD/locally-uncensored; COMPUTER-CONTROL.md documents every prompt and every restriction, and the changelog has a complete "what was tested, what wasn't" list.

For people focused on local AI and privacy, the value of this project isn't how flashy "letting the model click the mouse for you" is, but that it makes permission boundaries a first-class citizen of the product: non-skippable confirmations, an explicit list of forbidden zones, and an interruption mechanism that lets you take over at any time. At a time when agent-type products are all chasing "one-click full automation," this conservatism is instead a rare design attitude. As for whether it's actually good to use, that depends on which model you use—and from the author's testing, a small, fast text model combined with a control list is currently more practical than a dedicated vision model.

One-line Summary

Locally Uncensored 3.0.6 adds controlled computer operation for local models, ensuring safety through layered prompts and a forbidden-zone list.

Paragraph Summary

Locally Uncensored 3.0.6 adds computer control, allowing local models to operate user-approved apps via mouse and keyboard, with all screenshots and clicks staying on the local machine. The feature is off by default, and once enabled it still requires confirmation at four points: every new conversation, first operation on an app, every action (supervised mode), and sensitive operations (any mode). Terminals, password managers, system dialogs, and other areas are permanently off-limits. The author's testing found that a text model relying solely on control lists was faster than a vision-capable model (381s vs 497s), and the dedicated screen model Holo 3.1 9B succeeded only once in three runs. Platform support is uneven: Windows is verified, macOS isn't thoroughly tested, and Linux hasn't been run yet. Overall, it's positioned as a beta feature requiring human supervision.