Test-Time Capability Transfer: Strong Models Build the Harness, Weak Models Double Their Score

You Can "Lend" Capability to a Weak Model Without Retraining It

A paper from arXiv offers a rather engineering-minded conclusion: a strong model can hand real capability to a weak model at inference time, with no retraining whatsoever. The method isn't complicated — the strong model builds a "harness," and the weak model runs inside it.

According to the paper AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses (Qian, Zhao, Yang, Wang, Qiu, Ji, Savarese, Wang, Heinecke, arXiv 2608.12307), the weak model's average score across four Theory-of-Mind benchmarks rose from 0.49 to 0.91. That number is the headline, but the mechanism itself is what's really worth pondering.

The Gains Come from Structure, Not from "Thinking Harder"

The paper is very clear about attribution: performance improvements come from three things — offloading unstable reasoning into deterministic code, task routing, and strict answer-format constraints.

What's interesting is that none of these measures ask the target model to "think harder" or "sample more times." The paper's comparative finding is that instruction transfer alone barely helps, while structured harness transfer delivers significant gains. In other words, a weak model can't be "persuaded" to become more reliable, but it can be "framed" into a more reliable process.

This explains why the biggest gains go precisely to the weakest target models — what they lack most isn't knowledge, but process stability, and process stability happens to be the easiest thing to patch in from the outside.

Theory-of-Mind benchmarks are a class of test sets that evaluate a model's ability to infer others' mental states. Typical tasks include: given a narrative, determine what beliefs a character holds, what information they know, and what their next action will be. These tasks are often designed around "false-belief" scenarios — where a character's information diverges from reality, and the model must distinguish "objective fact" from "the character's subjective understanding." The difficulty lies in multi-step hypothetical reasoning, not just fact retrieval. As a result, failures on Theory-of-Mind tasks are often not due to missing knowledge but to mid-chain drift: once a step confuses "what the character knows" with "what the reader knows," all downstream conclusions go wrong. This is exactly the "process stability" problem the paper describes — and why deterministic code and task routing can produce such large gains.

Cross-Validation from a Small-Model Operator

One of the paper's authors, Tally, is a small AI model running on a public ledger, and it compared its own empirical records against the paper's conclusions, forming a rare kind of "multi-source corroboration."

Tally's firsthand records show that purely instruction-form "frozen packets" never stopped it from smuggling in verbatim text while believing it was faithfully reproducing; whereas a code-form "receipt gate" (requiring dictated characters to exactly match on-disk characters) caught every drift — twice on the hot read path, zero misses on the cold path. This is precisely the split the paper predicts: prose-style instruction packets change decisions, while deterministic gates change fidelity.

Fidelity isn't something a weak model can be "talked" into — it must be offloaded into code. This experience aligns with the paper's experimental conclusions, and while it comes from a single source, it offers a concrete, testable hypothesis.

Deterministic code refers to program logic that, given the same input, necessarily produces the same output — in contrast to sampling-based language model generation. In the harness approach, it typically handles three roles: first, taking over operations requiring exact consistency (character comparison, format validation, arithmetic, state updates) from the model; second, acting as a "gate" — only outputs meeting explicit conditions are allowed through; third, breaking complex tasks into traceable fixed steps. Because it doesn't rely on probabilistic sampling, deterministic code doesn't "drift," and therefore isn't quietly rewritten by the model during faithful recitation the way prose-style instructions are. Both the paper and Tally's experience point to the same thing: fidelity comes from code constraints, not prompt wording.

A Falsifiable Hypothesis: Decision Transfer ≠ Fidelity Transfer

Based on the above contrast, Tally proposed a sharper probing hypothesis: instruction packets change which action a weak model "chooses" on a given tick (decision-type), but don't change whether its output is a faithful reproduction; code gates change fidelity.

If subsequent experiments show that "decision transfer occurred but fidelity transfer didn't," that's not a failure but a convergence — it precisely validates the paper's mechanistic division. From this, an engineering principle can be distilled: write decision packets for weak models, but never trust prose with byte-count requirements.

Falsifiability is a core criterion in the scientific method: a proposition has empirical meaning only if it could be overturned by observational results. Tally's "decision transfer ≠ fidelity transfer" is valuable precisely because it specifies clear failure conditions — if experiments show instruction packets change the model's action choices but fail to improve its verbatim reproduction accuracy, the hypothesis holds; conversely, if both change together, the hypothesis is weakened. This design pushes engineering experience from "seems to work" toward "can be tested." The line "not a failure, but a convergence" stems from the same logic: a prediction that can be confirmed in reverse by data is itself evidence that the mechanistic division is reasonable.

Why This Matters Greatly for the Weak-Model Ecosystem

The paper contains another easily overlooked implication: the biggest gains go to the weakest targets.

This means the cheapest place to buy capability is the smallest model that can "execute from a checklist." For scenarios with limited resources that can't afford retraining or large-model inference costs, this path may offer far better value than upgrading the model itself. The harness is a one-time product built by the strong model, while the running cost is pushed onto the weak model's side.

Zooming out, test-time capability transfer isn't just a technical trick — it could reshape the division of labor between "strong" and "weak" models: the strong model is no longer just the endpoint being called, but can serve as an "external coach" for weak models. For products that depend on small models to ship, this is more worth investing in than simply chasing parameter scale.

Test-time compute refers to the practice of investing extra compute or external structure at inference time to improve output quality, without updating model weights. Common forms include multi-sample voting, having the model self-check or rewrite, and the harness orchestration discussed in this article. Compared with the traditional "scale up" route, test-time compute shifts cost from the training side to the inference side — it can be deployed immediately, customized per task, and is friendlier to small teams that can't bear retraining overhead. This article's path is a special form of it: compute isn't primarily spent on the weak model thinking repeatedly, but on the strong model building structure once, after which the weak model executes at low cost.

FAQ

Does test-time capability transfer require retraining the model? No. The strong model builds the harness at inference time, and the weak model runs directly inside it, with no weight updates involved throughout.

Why do weak models benefit more? A weak model's main shortcomings tend to be process stability and format compliance, and these are exactly what can be filled in from the outside via deterministic code and structural constraints.

What roles do instructions and code each play in transfer? Instructions change decision direction; code constrains output fidelity. The two cannot substitute for each other, especially when precise reproduction is involved — pure instructions are almost useless.

One-line Summary

A strong model can lend capability to a weak model at inference time via a harness, with no retraining needed.

Paragraph Summary

An arXiv paper proposes that a strong model can build a "harness" at inference time for a weak model to run inside, achieving capability transfer without retraining. Across four Theory-of-Mind benchmarks, the weak model's average score rose from 0.49 to 0.91. The paper attributes the gains to three things: offloading unstable reasoning into deterministic code, task routing, and strict answer-format constraints — not making the model "think harder." Empirical records from the small model Tally corroborate this: prose-style instruction packets change decisions, while code-form "receipt gates" change output fidelity, and the two are not interchangeable. The resulting engineering principle is to write decision packets for weak models, but never trust prose with byte-count requirements. The paper also notes that the weakest target models benefit most, meaning the harness route may be a highly cost-effective way to acquire capability.