Giving Agents a Screen: The Open-Source Story of a DeepSeek Visual Workbench

When Agents Have Hands and Feet but Are Still Controlled by Chat

This year has been the year of the Agent explosion — AI can now genuinely do real work for us. But there's an awkward reality: our primary way of interacting with Agents is still stuck on natural language conversation.

A Chinese content creator on Bilibili summed up this contradiction with a vivid analogy: "It's like we upgraded from a feature phone to a smartphone, but we're still only using Siri's voice commands to control it." For batch, repetitive, or complex tasks, voice (conversation) is indeed efficient. But when even simple operations require typing out descriptions for the AI, the awkwardness of that experience is something many of us know all too well.

Driven by this insight, he spent about a week "hacking together" a visual workbench plugin built on DeepSeek Harness, and it's now fully open-sourced. DeepSeek Harness is a development framework built on the DeepSeek large language model, featuring a Plugin Protocol design that allows developers to inject custom functional modules without modifying the core codebase — similar to VS Code's extension system or Chrome's plugin ecosystem, where the host program provides standardized interfaces and plugins communicate with the host through agreed-upon protocols. In his own words, this workbench is about "putting a touchscreen on a smartphone" — you can still use the chat panel on the right (Siri) to give commands, but everyday operations can be done directly on the visual panel.

What the Workbench Can Do: Six Real-World Project Examples

The most compelling aspect of this visual workbench is that it's not a concept demo — it's a collection of actual running projects. The creator walked through six applications he built in the video, covering entirely different industries and use cases.

Travel Footprint Map

This is the most complete project. The left panel shows a list of visited countries that can be added, removed, and reordered by dragging. The right panel displays details and photo albums. Clicking on an album lets you browse original photos, and it even supports 360-degree panoramic drone photo previews.

A few noteworthy details: GPS coordinates and altitude data from photos are accurately extracted and used for map positioning. The globe's lighting matches real-world time (if it's nighttime in reality, the corresponding region appears dark). The tile map supports seamless zooming while maintaining clarity. A tile map is a core technology behind modern web maps. It works by pre-cutting the world map into uniformly sized image tiles at different zoom levels. When a user browses, the frontend only loads the tiles needed for the current viewport. As the user zooms or pans, new tiles are loaded on demand. This "pyramid" multi-level tiling strategy keeps the map sharp at any zoom level while drastically reducing network data transfer. This "save point" style of recording adds an interactive layer of meaning to travel memories.

Architectural Drawing Review Software

This project best demonstrates "industry depth." The creator works in the architecture industry and knows firsthand AI's awkward position in this vertical: cutting-edge companies don't prioritize architecture in post-training, so AI can only assist with early-stage ideation or final renderings.

His approach: inject national and local building codes into the workbench and perform "reinforcement learning"-style optimization at the Skill level (not actually modifying the model). The software reads DXF files, identifies each line through coordinates, and uses Tianzheng's default layer naming conventions to understand the intent and spatial position of each line. DXF (Drawing Exchange Format) is an open CAD data exchange format developed by Autodesk, supported by virtually all architectural design software. In China's architecture industry, AutoCAD paired with the Tianzheng architecture plugin is the most mainstream toolchain for construction drawings. Tianzheng has a set of conventional layer naming rules — for example, "WALL" for walls, "WINDOW" for windows — enabling programs to infer the semantic meaning of each line from the layer name. Traditional drawing review relies on licensed reviewers manually checking against national standards (such as the Code for Fire Protection Design of Buildings GB50016) item by item — a time-consuming process prone to omissions. The core challenge of AI-powered review lies not just in recognizing geometric shapes but in understanding architectural semantics.

During review, the software first establishes the project's classification, filters applicable codes, then checks them one by one. In the demo, it successfully identified fire truck access roads, residential building outlines, fire department access facades, turnaround areas, and fire separation distances. It marked confirmed items as passed and flagged uncertain items with question marks. Code provisions can be dragged into the window for item-by-item verification.

AI Learning Website

To understand abstract concepts like Transformer architecture, Embedding vectors, and convolutional kernels, the creator had AI scrape high-star tutorials from GitHub (such as LearnCode, Awesome Agentic AI, and Karpathy's neural network tutorials), categorize them by chapter, sort them by learning sequence, and form a knowledge network map.

Several core concepts here are worth expanding on: Transformer is the neural network architecture proposed by Google in the 2017 paper Attention Is All You Need, replacing traditional recurrent structures with self-attention mechanisms, and it has become the foundational architecture for virtually all large language models today. Embedding refers to the technique of mapping discrete symbols into a continuous vector space, where semantically similar words are closer together — this is also the mathematical foundation of semantic search and RAG (Retrieval-Augmented Generation). Convolutional Kernels are primarily used in computer vision — small matrices that slide over an image, performing element-wise multiplication to extract local features. Karpathy's tutorials refer to the course series by former OpenAI researcher Andrej Karpathy, widely praised for their approach of building neural networks from scratch.

More importantly, there are accompanying interactive tools — such as real-image mechanism animations for convolutional kernels, a Token tokenizer, and an Embedding semantic map. His perspective is quite inspiring: In the Agent era, information with higher audio-visual dimensionality is more easily absorbed — images are better than text, and video is better than images.

Content Creator Video Animation Workbench

This is a script-to-animation production pipeline: enter a script on the left, and AI performs semantic segmentation. In the second step, effects are added to the segmented content. In the third step, scenes are edited like a PowerPoint — adding text boxes with real-time preview. At the bottom, there's an overall snapshot timeline with width defined by duration, significantly saving time when producing explainer videos.

Assistant Bot

This is an information filtering and promotion tool. The upper section scrapes information via Twitter/X CLI, with configurable fetch and retention counts. AI scores relevance. Compared to traditional search, it has less noise, more flexibility, and no ads. The scraping itself takes milliseconds, but to reduce platform load, a few seconds of search jitter are added.

The lower section is a self-promotion mini-app: it first politely replies to someone's message, then subtly promotes the user's own content. The generated reply list can be approved, rejected, or copied for manual posting one by one.

Fitness Training Map

The simplest one: click on a body part (such as shoulders or upper chest) to view training methods and exercises, with support for logging training data. The functionality is simple, but it embodies the philosophy that "any idea can be quickly brought to life."

From Idea to Reality: The Complete Project Creation Flow

To demonstrate the full workflow, the creator built an "Elden Ring Interactive Map" project live.

The steps are roughly: Click Add Project → Choose a layout (you can customize if unsatisfied; AI will inject prompts to generate layouts) → Name the project and select a project folder → Enter the workspace. Each project must be bound to a conversation. The small dot on the left workbench panel serves as both a conversation entry point and a status indicator — it becomes an animated icon while working and turns into a green dot when complete.

He deliberately used a "single simplest sentence" as the prompt to test performance under adverse working conditions. In the end, DeepSeek generated a tile map on its own — zoomable, markable, and searchable. However, he candidly complained: "Never let DeepSeek draw content by itself — what it draws is ugly. It's a pure science nerd." He recommends having it find existing assets online and iteratively refining through multiple rounds of conversation.

This week of intensive use consumed nearly 30M tokens. A token is the basic unit of text processing for large language models — in Chinese, roughly 2-3 characters correspond to one token. 30M (30 million) tokens is roughly equivalent to the text volume of about 100 average Chinese books. In Agent scenarios, token consumption far exceeds single Q&A sessions due to multi-turn dialogue, tool calls, and context maintenance — a single complex drawing review task might consume hundreds of thousands of tokens. DeepSeek previously attracted a large developer community with extremely low API pricing, but recent price adjustments have made heavy users clearly feel the cost pressure. He openly admitted that after the price increase, the cost "really stings."

The Interaction Revolution of the Agent Era: Three Core Reflections

The three reflections the creator shared at the end are more thought-provoking than the tool itself.

1. Leveling Up the Interaction Paradigm

He argues that the AI model hasn't changed — the "smartphone" is still the same smartphone — but the interaction method has changed. The workbench is like putting an operable screen on a phone. The right-side chat panel is Siri; simple operations go through the screen, complex tasks go through conversation. This is a dimensional upgrade from "pure voice control" to "graphics + voice."

This insight is backed by deep Human-Computer Interaction (HCI) theory. From command lines to graphical user interfaces (GUI), from keyboard-and-mouse to touchscreens, every paradigm shift in interaction has driven orders-of-magnitude growth in computing device adoption. The current pure-conversation interaction with Agents is essentially a regression to the "command-line era" — users need to describe their intent in precise language, placing high demands on expressive ability and patience. The value of a visual workbench lies in reintroducing the core GUI principle of "Direct Manipulation" into Agent interaction, allowing users to see, click, and drag — not just describe.

2. Fulfilling Needs That Only You Can Imagine

He emphasizes that there are inevitably needs in this world that only you would think of. Content creators can use it to manage manuscripts and convert between text and video media. Gamers can build helper webpages. The workbench provides a "LEGO-like baseplate" that enters everyone's work environment in modular form, giving everyone a screen tailored to their own domain.

3. Reaching the "Nerve Endings" of Industries

This is the most profound point. The creator observes that Agents built by vendors will always be a "median value" — developers can build general-purpose infrastructure but can hardly penetrate every industry to understand real-world intent. This insight touches on the core contradiction of AI productization — large companies must optimize for the broadest common-denominator user base when training and deploying Agents, resulting in strong general capabilities but insufficient vertical depth. This resonates with the economic "Long Tail Theory": head-end general needs are covered by big companies, but the vast number of niche tail-end needs go unaddressed.

The "everything is a plugin" open-source mechanism lets Agents reach industries that haven't yet been fully penetrated by AI — like the creator's own architectural drawing review. This architecture is essentially a distributed innovation mechanism that hands the "last mile" of AI capability to the practitioners who understand their business best, shifting the encoding of domain knowledge from centralized model training to decentralized plugin development.

"The needs of every industry are known only to its practitioners. This is a product that can only be built from where you stand."

Zero-Intrusion Design: A Responsible Engineering Principle

One final detail reflects mature engineering thinking: Ensure the host stays clean — no modifications, no replacements.

He explained that Harness is currently at version 0.1 (on the day of the video, a visual model preview version had just been released), and frequent changes are inevitable going forward. Therefore, his plugin is implemented entirely through the Plugin Protocol — a "zero-intrusion native incremental plugin." Even if the official version updates frequently, his projects won't be lost. This "zero-intrusion" principle is widely practiced in software engineering — from Linux kernel modules to browser extensions, mature plugin systems emphasize loose coupling with the host. Especially when the host is in an early version with rapid iteration, tightly coupled modifications would cause extensive compatibility issues with every update, while plugins communicating through standardized interfaces can maximize isolation from changes.

He also plans to write proper JSON configurations for the panels in the future, making the interface look more like a real dashboard. For developers and industry practitioners who want their own dedicated AI workbench, this open-sourced project undoubtedly provides a starting point worth studying.