Text-Only Models Controlling a Linux Desktop: Inside PawFlow's Dual-Model Architecture

A Counter-Intuitive Experiment: Letting a "Blind" AI Operate a Desktop

When people think of AI Agents that can control a computer, most assume they must rely on multimodal large models with vision capabilities. An AI Agent, broadly defined, is an intelligent system that perceives its environment, plans autonomously, and executes actions to achieve a goal. In desktop automation, mainstream approaches — such as Claude Computer Use or GPT-4V-powered agents — typically depend on multimodal models that can simultaneously process images and text, directly "seeing" the screen end-to-end and deciding what to do next. This feels intuitively natural, but comes at the cost of expensive API calls and strict rate limits.

The open-source project PawFlow offers a fundamentally different answer: letting a text-only model control a full Linux desktop environment — while never being able to "see" the screen.

The system's core configuration is cleverly designed. The reasoning model is glm-5.2:cloud, running on Ollama's free cloud tier. Ollama is a popular local LLM runtime that lets developers run open-source models on their own hardware or Ollama's hosted service, without depending on commercial cloud APIs. The GLM (General Language Model) series is jointly developed by Tsinghua University's KEG Lab and Zhipu AI; GLM-5.2 is a lightweight inference variant focused on text understanding and logical reasoning, with no vision capabilities. The "eyes" of this text-only model are provided by a second, independent vision model. The entire solution is self-hosted, MIT-licensed, fully open to developers, and runs at near-zero cost.

The demo video the author shared runs about 70 seconds. The instruction given to the Agent is a single sentence: "Open the browser, go to YouTube, search for Linkin Park's 'Burning in the Skies,' and play it." The Agent proceeds to open Chromium from the application menu, completes the search, handles a consent popup, clicks the correct result, and finally takes a screenshot to verify playback — all smoothly, and entirely on the free tier.

Dual-Model Collaboration: Fully Decoupling "Seeing" from "Thinking"

The most fundamental design in the PawFlow architecture is decoupling visual perception and semantic decision-making into two separate models. This decoupling philosophy draws from classic modular system design, long practiced in robotics: the vision model only needs to excel at image understanding and spatial localization, while the text model only needs to excel at natural language reasoning and task planning. Each works within its own area of strength, avoiding the predicament of demanding a single model to simultaneously master two fundamentally different tasks. This also aligns with the Mixture of Experts (MoE) design philosophy — having specialized modules collaborate often outperforms having a single generalist model handle everything.

Vision Model: Responsible for Describing the World

Every screenshot is first sent to the vision model, accompanied by a prompt requiring detailed description. It outputs the following structured information:

• Visible text (OCR results)
• Overall page layout
• UI controls and their approximate [x, y, w, h] pixel coordinate bounding boxes
• Current state of each element

Text Model: Responsible for Decision and Action

The text model (glm-5.2) receives not the image itself, but the structured textual description above. It makes operational decisions based on that description — which button to click, which input field to type into.

This decoupling delivers a key advantage: the entire pipeline requires no expensive multimodal API calls and can run entirely on the free tier. For budget-constrained developers and independent builders, this is an extremely attractive self-hosted desktop automation solution.

Click Precision: Copy Coordinates, Don't Guess Them

The author admits that click precision was the most surprising highlight.

Traditional vision agents typically require the model to "estimate" click coordinates, which is often a major source of errors. PawFlow's design completely sidesteps this problem: the text model never guesses coordinates. It is only responsible for identifying the target element from the vision model's description, then directly reusing the coordinate values already computed by the vision model.

Coordinate generation and coordinate usage are separated across two models: the text model handles "semantic judgment" — understanding which element is the target — while the vision model handles "spatial localization" — outputting precise pixel coordinates. This division of labor turns out to be more reliable than asking a single model to handle both tasks simultaneously.

Pre-Click Guard: Local Image Diffing to Prevent Mis-Clicks

Accurate coordinates don't guarantee a foolproof outcome. When the interface changes dynamically — a page refresh, a popup suddenly appearing — coordinates may become stale before they can be executed. PawFlow addresses this with a pre-click guard mechanism.

Before executing a click, the system re-captures a partial screenshot of the target area and performs an image diff comparison against the screenshot that originally generated those coordinates. Image differencing is a foundational computer vision technique: it compares pixel values of the same region at different moments, and flags any difference exceeding a threshold as a "change." This technique is already widely used in video surveillance and UI automation testing (such as visual regression testing in Selenium). It has extremely low computational overhead — no GPU or model inference required. PawFlow applies it as a pre-action verification step, essentially borrowing a well-established software engineering stability technique to compensate for AI decision-making's temporal blind spots. This step runs entirely locally and consumes zero LLM calls.

If a change in screen state is detected, the click is cancelled, and the Agent re-captures and re-analyzes the screen. This mechanism effectively prevents erroneous actions based on outdated information, significantly improving the robustness of the automation pipeline.

Description Caching: Handling Free-Tier Rate Limits

The biggest pain point of a free cloud tier is rate limiting. PawFlow employs a simple yet effective countermeasure: caching visual descriptions by image content hash.

Content-addressed caching is a strategy that uses the hash of data content (such as MD5 or SHA-256) as the cache key. It is widely used in CDNs, package managers (such as npm and pip), and build systems (such as Bazel). Its core advantage is that identical content always produces an identical hash, enabling natural deduplication without manual cache invalidation management. PawFlow applies this mechanism to screenshots: if the screen pixels haven't changed, the hash remains the same, the cache is hit, and the vision model call is skipped entirely.

When the Agent visits the same screen state again, it directly reuses the cached description, bringing additional vision call costs to zero. This optimization is especially critical in rate-limited environments: it conserves precious API quota, speeds up the Agent's response on repeated interfaces, can multiply effective usage time, and is a key safeguard that keeps the entire solution running reliably on the free tier.

"Composition Over Monolith": Architectural Insights

PawFlow offers a thought-provoking perspective: the capability ceiling of an AI Agent may not be determined by the limits of any single model, but by the quality of the system architecture's design.

By modularizing visual description, semantic decision-making, coordinate localization, safety guards, and result caching into separate components, a text model that "can't see" can still reliably operate a graphical interface. This hybrid "AI + traditional engineering" architectural philosophy — using proven software engineering techniques (diff detection, content-hash caching) to shore up AI's weaknesses — offers direct reference value for AI Agent development in resource-constrained environments.

The project is open-sourced on GitHub (PawFlow-Agents, MIT license) and includes a detailed configuration guide for this demo (GLM 5.2 + vision fallback solution). For developers looking to explore self-hosted desktop agents, this is both an open and pragmatic starting point.

Note: This article is based on a single source — the author's Reddit post and demo video. Actual stability and generalizability still await broader validation across different users and task scenarios.

Key Takeaways