Harness Makes or Breaks It: How Codex CLI Transformed a Local LLM

Same Model, Wildly Different Results

A Reddit user's hands-on experience has once again confirmed the old adage in the AI coding community — "Harness matters." He runs an OpenAI subscription alongside locally deployed LLMs and has also tried DeepSeek^Ad's latest flash version. His hardware setup is seriously capable: dual 3090s with ample RAM, running Qwen 3.8 Flash Next at impressive speeds.

The problem? His local LLMs had always been little more than demo toys. Beyond his go-to benchmark — "build me a multi-level 3D Mario game" — whenever he tried doing real, serious work, local models never came close to GPT 5.2, or even the later GPT 5.6 Luna despite its lower benchmark scores.

The Turning Point: Swapping the Harness, and Suddenly It Clicks

Everything changed when he asked GPT 5.6 Luna to help him configure Codex CLI to work with his local LLMs. He had previously been using pi.dev and opencode as his execution frameworks, with disappointing results. Switching to Codex CLI left him genuinely stunned.

In his own words: "The AA benchmarks suddenly started making sense." His first test was the same 3D Mario prompt — and the result from Codex CLI was the best he had ever seen.

This detail deserves every local-model developer's attention: the model weights didn't change, the hardware didn't change, the prompt didn't change. The only thing that changed was the execution framework (harness), and yet output quality made a qualitative leap. It's a clear signal that how good a model "feels" depends heavily on what kind of toolchain it runs inside.

Harness (execution framework) in AI coding refers to the full toolchain wrapping a model's calls — covering context management strategy, system prompt structure, tool call scheduling logic, multi-turn conversation state tracking, and interaction protocols with the filesystem, terminal, and external APIs. Even with identical model weights, different harnesses can produce radically different prompt structures, token budget allocations, and error retry mechanisms. Codex CLI is OpenAI's developer-facing command-line AI coding assistant; its core design philosophy grants the model direct read/write and execution access to local codebases, while structurally decomposing tasks to significantly improve completion rates on multi-step coding jobs. pi.dev and opencode are two other agentic coding frameworks with similar goals but different implementation approaches, each making different trade-offs around context compression and tool-calling strategies. These framework-level differences often have a more direct impact on real-world coding experience than model parameter count.

Head-to-Head: Qwen 3.8 Flash Next Beats Luna

The real proof came from an actual project. The author handed a project that Luna had been working on for several days to both setups in parallel. The result was surprising — Qwen 3.8 Flash Next running through Codex CLI left Luna in the dust.

The historical context makes this even more striking: on that same project previously, both Qwen and DeepSeek had failed to deliver. Now, the author flatly calls Qwen 3.8 Flash Next his go-to model.

The value of this case isn't some sweeping conclusion that "local models crush closed-source ones" — the sample size is small and quantitative reproducibility is lacking. What it does reveal is a commonly overlooked variable: the same local model under different harnesses can span the entire range from "completely unusable" to "production-ready first choice."

Web Search: Porting pi's Capabilities as a Codex Skill

The author also shared a practical engineering detail: to give Codex CLI web search capability, he had the model port the pi-smart-web-search feature from the pi ecosystem into a Codex skill (plugin), reporting excellent results and plans to open-source it on Git.

This move is itself an extension of the "harness matters" principle — the capability ceiling of an execution framework is largely defined by its extensible skill/plugin ecosystem. Codex CLI's skill mechanism lets users migrate the best features from other tools, continuously filling in gaps.

pi-smart-web-search is a built-in web retrieval feature in the pi.dev ecosystem that can fetch and summarize web content in real time during a conversation, allowing the model to reference the latest documentation or Stack Overflow posts instead of being limited to its training data cutoff. Porting it as a Codex CLI skill means writing a wrapper function using Codex's tool registration mechanism: defining the function signature (input: query string, output: summary text), handling HTTP requests and result parsing, then exposing it to the model via Codex's tools configuration. This kind of skill migration is capability transfer between agentic frameworks — since most frameworks' underlying tool-calling protocols are based on the OpenAI function calling spec, cross-framework porting is feasible in principle, though it still requires adapting to each framework's sandbox permissions and call sequencing.

An Open Question: Was pi.dev Being Used Wrong?

The author ends his post with admirable honesty and openness: "Maybe I was using pi.dev wrong. Maybe there's an extension that brings it to the same quality as Codex CLI. Does anyone know?"

This is a completely fair question. A single user on a single project can't rule out confounding factors like configuration differences, parameter settings, or context management strategies. The fact that Codex CLI won in his testing doesn't necessarily mean the framework is categorically superior — it may simply be that its default configuration suits local model invocation better, or that pi.dev and opencode require additional tuning to perform at the same level.

Key Takeaways for Developers

Setting aside the limitations of a single anecdote, the signals this post sends are quite useful for local LLM users:

• Don't judge a model solely by its benchmark scores. A model with higher benchmarks (like Luna) isn't necessarily faster or more reliable in real projects — the execution framework dramatically shapes the final experience.

• Switching harnesses is a low-cost, high-leverage move. Without changing hardware or retraining any models, trying different CLI/agent frameworks can directly determine whether a local model is actually capable of doing real work.

• Skill/plugin ecosystems are a key competitive advantage. Whether capabilities like web search can be easily ported and extended directly affects the practical ceiling of a framework.

For anyone running consumer or prosumer hardware like a 3090 and looking to put local models into real production use, it's worth treating "dialing in your harness" as just as important as "picking your model."