DeepSeek Harness vs Pi: Six-Round Head-to-Head Test of Two Free Coding Agents

When you want a coding agent that isn't from Anthropic or OpenAI, Pi and DeepSeek Harness (DSH for short) are nearly the only two mature options. As it happens, within roughly a single day, Pi released version 1.0 and DSH launched its desktop app. Both are MIT-licensed and completely free — you only pay for the model itself. This article is based on the largest same-model benchmark comparison to date, breaking down their real differences across six dimensions.

The Framework War: Minimalism vs. Everything-as-a-Plugin

To understand these two tools, you first need to understand what a "framework" is. A framework is everything around the model — the loop that keeps it running, the tools it can call, the system prompt, and how it manages the context window. How much does the framework matter? In one editing benchmark, the same GPT model's score dropped from 99% all the way to 70% — with the framework being the only variable.

Pi and DSH bet on opposite ends of this spectrum. Pi is a minimalist agent built by Mario Zegner, now operated by a company called Arandio. It gives you exactly 4 tools — read, bash, edit, and write — with a system prompt only a few hundred words long. Everything else requires extensions. Its loyal users call it "the Vim of coding agents."

DeepSeek^Ad Harness goes to the other extreme, with the philosophy that "everything is a plugin" — even the agent loop itself is a plugin. Its default configuration calls 26 tools on the very first request. One is minimalist to its core; the other is loaded out of the box. This fundamental difference shapes how they perform in every round that follows.

MCP (Model Context Protocol) is an open protocol proposed by Anthropic in late 2024, designed to standardize how AI models communicate with external tools and data sources. Think of it as the "USB port for AI tool calls" — any tool that implements an MCP server can be called by any framework with an MCP client, without custom adapters for each model. Both Pi and DSH support MCP, but Pi 1.0 goes further with a "code mode" that lets the model invoke tools by writing JavaScript code within a sandbox rather than calling individual MCP tools one at a time. This is significantly more efficient for tasks that require loops or aggregation — which is exactly why the car safety task in this article shows such a dramatic cost difference.

Round 1 · Installation: DSH's Desktop App Is More User-Friendly

DSH offers a desktop app for Mac and Windows that bundles everything you need. Install it, sign in with a DeepSeek account, and you're ready to go. It's essentially a wrapper around DSH's existing web interface, so there's still no terminal UI. Linux users are limited to the NPM version.

Pi lives in your terminal: install it via NPM or an installer, configure an API key or point it at a local model, and start typing. If you already live in the terminal, it feels like home; if you're not comfortable with the command line, DSH's desktop app is clearly friendlier. Round 1 goes to DSH.

Round 2 · Same-Model Performance: Essentially a Tie

This is the round everyone asks about — with the same model, which one is better? The answer is basically a draw.

In terminal tests using DeepSeek V-Flash, Pi passed 66 out of 89 tasks while DSH passed 65 — and DSH only hit that number after testers fixed three configuration issues (including a 5-minute timeout on tool calls). On 30 real-world tool-use tasks, Pi passed 21 and DSH passed 20, with the two agreeing on 27 of the 30 outcomes.

On the Kimi model, DSH's plugin-builder mode edged Pi by one task; default mode was a tie. DSH was faster. Worth noting: almost all these tests were run on a pre-1.0 version of Pi, and no one has re-run them since. Also worth a reality check: OpenAI's Codex outperforms both in both comparison tables. Round 2 is a draw.

Round 3 · Cost: Pi's Minimal Prompt Is Cheaper

The cost difference comes directly from their prompt strategies. Pi keeps its prompts as small as possible; DSH sends a large prompt — in one measurement, about 10× larger than Pi's — and relies on caching to offset the cost.

This works well on DeepSeek's own API, where cached tokens cost only 1/50th the price of new tokens. But on Claude Sonnet, cached tokens cost 1/10th the price of new tokens, which makes a large prompt far more expensive. In testing, Pi cost just over 2 cents per passing task, while DSH ran slightly higher. On Kimi, Pi's median cost was $2.43 versus DSH's $3.46.

If you stick with DSH, one practical tip is to turn down the reasoning intensity — research suggests this can cut costs by roughly three-quarters while still passing each task. Round 3 goes to Pi.

"Prompt Caching" is a mechanism offered by major API providers to reduce costs for repeated content: when the prefix of a request matches a recent one, providers charge full price only for the new, uncached tokens, with a discount on the cached portion. DeepSeek's API offers an especially aggressive caching discount (roughly 1/50 of new token cost), which makes DSH's strategy of "sending a large system prompt and relying on caching" nearly free on DeepSeek's own platform. But when users switch to providers like Claude — where the discount is smaller (roughly 1/10) — the cost advantage of large prompts quickly disappears. This explains why the article specifically distinguishes cost performance across different API platforms.

Round 4 · Model Support: Pi Is More Open

Pi can talk to roughly 30 providers and has a built-in router for connecting to local models via llama.cpp. Pi 1.0 also introduced "virtual models": you select a model from the menu, and an extension decides at runtime which real model actually handles each request. In Arandio's release demo, Pi even wrote itself an extension that plans with Claude Opus and writes code with GPT.

DSH defaults to DeepSeek's own Flash model. You can add Anthropic, OpenAI, Kimi, or any compatible endpoint — but interestingly, this entire third-party layer actually runs on top of Pi's model library (a package called DSLLM), and only DeepSeek's own models have native adapters. It's not surprising that a model lab's framework would favor its own models by default. For local models, Pi supports llama.cpp, Ollama, and vLLM, with more configuration options overall. Round 4 goes to Pi.

Round 5 · Out-of-the-Box Features: DSH Is More Full-Featured

This round compares what you get before installing anything extra. DSH wins decisively. Its defaults include planning mode, a to-do checklist, web search, workflows, and sub-agents — which can even hand off work to Codex or Claude Code. In creation mode, you simply request a new plugin in chat and the agent writes and installs it for the current session. DSH also launches each session in a sandbox, restricts write permissions, and asks before taking action.

Pi intentionally gives you only four tools and no sandbox — it explicitly recommends running it inside a container. Planning mode and sub-agents require community packages. Pi 1.0's major addition is MCP "code mode": the model writes a small JavaScript snippet that calls tools within a sandbox and returns only the result. One creator ran the same car safety task both ways — traditional MCP cost 26 cents and ranked only 4 cars; code mode cost 9 cents and ranked all 10. This explains why Pi dropped its old tagline of "no MCP." But for overall out-of-the-box experience, Round 5 still goes to DSH.

Round 6 · Control and Privacy: Pi Is More Open

For a tool that calls itself "open," control matters enormously. Both use MIT licenses, but only Pi genuinely accepts outside contributions: it has 290 contributors and merged 68 PRs in the past month. DSH has more than twice as many GitHub stars as Pi, yet only 42 contributors — and accepts zero external PRs. It's still in developer preview; the README warns in capital letters that breaking changes will occur, and 25 versions were released over 7 weeks. Some users report needing to re-check their plugins every 3–4 days.

Privacy is another key difference. By default, DSH sends your session logs to DeepSeek's own API with every request (this can be disabled in settings), and the desktop app reports usage analytics (documentation states there are no user-facing controls for this). Pi sends only anonymous install and update signals, with analytics off by default. More importantly, regardless of which framework you use, if you go through DeepSeek's API, its privacy policy explicitly states that your data is stored in China and may be used for training. Round 6 goes to Pi.

The MIT License is one of the most permissive open-source licenses in software, allowing anyone to freely use, modify, and distribute code — including for commercial purposes — with the only requirement being that the original copyright notice is retained. Both tools adopting the MIT license means their source code is public and user-auditable. However, the article reveals a meaningful gap between being "open source" and "accepting community contributions": DSH's code is visible but accepts no external PRs — it's a read-only kind of openness. Pi, by contrast, has an active contributor ecosystem. For enterprise users, the more important consideration is runtime privacy: a framework being open source doesn't mean the API services it connects to by default are under your control.

Conclusion: Which One Should You Install This Week?

Final score: Pi 3, DSH 2, plus one draw. The right choice depends on how you work:

• Choose DSH if you want a ready-to-go desktop app and you're running DeepSeek models on the DeepSeek API. Remember to turn down the reasoning intensity, and disable session logs if that makes you uncomfortable.
• Choose Pi if you live in the terminal, frequently switch between models, or run models locally. But before you hand it a code repository, give it a container first.

Both tools see new developments every few days, and next month's winner could be a third option entirely. The most pragmatic move is to save your agent instructions and skills as plain files in your repository — that way, switching frameworks takes only an afternoon.