Same Model, 29% Score Gap? How Multi-Harness Reinforcement Learning Is Breaking Open the Open-Source Bottleneck

Why the Same Model Can Perform So Differently

A puzzling phenomenon is troubling AI Agent developers: the same model, with the same weights, can score 62% in one agent harness (agent execution framework) and drop to just 33% in another. This nearly 30-percentage-point gap exposes a core pain point for open-source models in real-world Agent deployments — model performance is highly dependent on the framework environment it runs in.

The Hugging Face team and researcher Adithya S K recently published what many are calling "one of the most practical RL write-ups of the year" — a multi-harness reinforcement learning guide that is fully open-sourced. It attempts to answer a critical question: how can open-source models perform consistently across a variety of coding agent frameworks, rather than excelling in just one?

The Core Idea: Leave the Framework Alone, Add a Proxy Layer

The elegance of this approach lies in its non-invasive design. Traditional methods often require custom modifications for each harness, but the idea here is different: don't touch the harness itself — instead, point it at a proxy, which then connects to the model.

The proxy's key capability is "speaking four API dialects" — it can simultaneously handle OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Gemini, the four API protocols commonly used by coding agents. The proxy records the exact token IDs and logprobs from vLLM sampling, and training is built on top of this real data.

This means that whether you're using Claude Code, Codex, or OpenCode, developers don't need to change a single line of code to plug them into the training pipeline. This decoupled design allows the same training method to be reused across any harness, dramatically reducing the cost of adaptation.

vLLM is the leading high-performance inference engine for large language models, known for its PagedAttention technology, which efficiently manages KV Cache and supports continuous batching. In reinforcement learning training, vLLM plays the "sampling" role — having the model interact with the environment to generate rollout data. The "exact token IDs and logprobs" mentioned here are critical: token IDs record exactly which token the model generated at each step, while logprobs record the log-probability of that choice. Both are essential inputs for policy gradient computation in mainstream RL algorithms like PPO and GRPO — without them, the trainer has no way of knowing "why" the model made a particular decision, and therefore cannot correctly apply reward signals to guide behavioral optimization. By intercepting and recording these low-level sampling details, the proxy layer makes the harness completely unaware that training is happening, achieving a clean decoupling between the inference side and the training side.

Experimental Results: The Advantages of Multi-Framework Joint Training

Experimental data demonstrates the real gains from multi-harness training:

• Training simultaneously across 4 harnesses boosted Liquid AI's LFM2.5-2.6B model from 42% to 54%;
• Thanks to a small bonus reward for completing tasks in fewer steps, tool call counts dropped by 31%;
• Training only on the OpenCode single framework pushed OpenCode scores from 34% to 58%, but models trained across multiple harnesses showed improvement across all frameworks.

This reveals an important trade-off: single-framework training can achieve higher scores in a specific environment but has limited generalization; multi-framework joint training, by contrast, delivers broad and robust improvement. For the real-world scenario where "an Agent will run in dozens of different harnesses," the latter is clearly more practical.

The "small reward for completing tasks in fewer steps" mentioned in the experiment is a reward shaping technique for sparse reward settings. The primary reward for coding agents is typically binary — the task either passes the test and earns 1 point, or it doesn't and earns 0 — making the reward signal extremely sparse and providing the model with little intermediate feedback to improve behavior. By introducing a "step efficiency reward," the trainer provides denser guidance signals at each intermediate step, encouraging the model to learn how to solve problems with shorter tool-call chains. The 31% reduction in tool calls not only means lower inference costs, but also lower latency and fewer failure points in real deployments — every unnecessary tool call is a potential point of failure.

Why Imitation Learning Falls Short: Practice Beats Copying

The team also validated a shortcut that many developers instinctively reach for: directly using successful trajectories from a larger model for supervised fine-tuning (SFT). They collected 3,189 successful rollouts from Qwen3.8-27B for imitation learning, and performance stalled at 47.5% — below the results achieved through two rounds of RL training.

This comparison is telling — simply copying a larger model does not enable a smaller model to truly learn how to solve problems. The key is reinforcement learning's approach of "repeated practice." The model needs to accumulate experience through real interaction and trial-and-error, rather than passively imitating someone else's answers. This also demonstrates that for highly interactive tasks like Agent deployment, RL's value cannot be replaced by supervised fine-tuning.

The fundamental difference between SFT and RL lies in the source of the learning signal. SFT is essentially behavioral cloning: show the model the "right answers" and have it maximize the predicted probability of those answers. For tasks like coding agents, the "right answer" is a successful operation trajectory — but the model doesn't understand why that trajectory succeeded, and it has never experienced failure paths. RL is different — the model tries things in a real environment, receiving positive rewards for success and negative feedback for failure, gradually learning a strategy through extensive trial and error. This distinction is especially pronounced for Agent tasks: an Agent must make continuous decisions in a dynamic environment, where a single mistake can cause everything downstream to collapse, and that kind of "chain failure" experience can only be gained through actual interaction — it cannot be learned from static successful trajectories. The 47.5% ceiling in the experiment is a direct result of the model lacking any awareness of failure modes.

Fully Open-Sourced: Anyone Can Train Their Own Agent Model

The most commendable aspect of this work is its thorough commitment to open source. From the capture proxy in OpenEnv and the trainer in TRL, to the task sets, SFT data, training code, and all seven trained models — everything is publicly available.

The significance of this openness is clear: Agents will run in dozens of different harnesses in the future, and now anyone can train open-source models for whichever framework they need, without being locked into closed-source solutions or expensive custom development. For teams looking to deploy small, efficient models within a specific coding agent ecosystem, this provides a reproducible and scalable path forward.

As Agent application scenarios become increasingly diverse, the ability to "make open-source models work across any framework" is becoming ever more important. This multi-harness RL guide not only provides a methodology — it places the complete toolchain directly in the hands of the community.