More Programs or More Repetitions? Decomposing Coverage and Specialization in LLM Harnesses

An Overlooked Evaluation Pitfall

Automatically generated LLM harnesses (reasoning execution frameworks) are often expected to improve large model reasoning performance through task specialization. The intuition is straightforward — customizing execution programs for different task types should outperform generic solutions. But a study published on arXiv raises a sharp question: can we actually prove that these custom programs deliver "useful specialization"?

The research team points out a phenomenon that's easy to conflate: additional answer coverage may simply come from repeatedly executing the same program, rather than from the program's specialized capabilities. In other words, if you run the same program several more times, the probability of hitting the correct answer naturally increases — and this gain is easily misread as a "specialization advantage."

LLM Harness refers to a programmatic execution environment built around a large language model, controlling input formatting, reasoning flow, output parsing, and answer extraction. In tasks like mathematical reasoning and code generation, researchers often automatically generate customized harnesses for different problem types — for example, generating an execution program with coordinate calculation logic for geometry problems — hoping that task-specific structural design helps the model "perform better." Oracle coverage is a commonly used metric for measuring harness collection performance: if at least one program in a set produces the correct answer on a given problem, that problem is considered "covered," and higher coverage theoretically indicates stronger collective capability. The problem is that oracle coverage is fundamentally an upper-bound metric — it only asks "can we hit it" without asking "how" — making it easy to misinterpret random hits from repeated execution as systematic gains from specialization.

The Clever Design of a Controlled Experiment

To disentangle the contributions from different sources, the researchers designed a controlled evaluation framework specifically to distinguish three effects: answer coverage, repeatable task advantage, and gains from pre-execution selection.

The experiment ran on 386 tasks from the MATH-500 dataset, comparing eight automatically generated harnesses plus one baseline program against nine byte-for-byte identical copies of the baseline as a control group. Each member was executed three times.

The "byte-identical copies" design is the key contribution of the paper. It serves as a clean control group — because these copies are exactly the same code, any score differences between them cannot possibly come from specialization; they can only arise from the randomness of execution. This provides a baseline for measuring "the gains from pure repetition."

How Much Can Repetition Alone Contribute?

The results show that identical programs under repeated execution produced a 2.16 percentage point improvement in average oracle coverage. In other words, simply running the same program more times squeezed out over two percentage points of coverage gains — gains entirely unrelated to any "specialization."

What Advantage Do Generated Programs Actually Offer?

The study found that automatically generated programs do exhibit more pronounced repeatable scoring patterns. At first glance, this looks like good news — stable patterns imply predictable behavior. But deeper analysis reveals the opposite conclusion: these repeatable patterns primarily expose persistent weaknesses, not persistent strengths.

The numbers are telling: 100 tasks showed performance losses relative to baseline that persisted across all three repetitions, while persistent wins appeared in only 1 task — and even that single win was highly sensitive to the answer extraction method used.

This means the generated harnesses were more consistently losing than consistently winning. Their repeatability doesn't translate into reliable task advantages.

Zero Gain from the Selector

The experiment also evaluated a "frozen selector" — a strategy that picks programs before execution based on some criterion. The result: 0.00 percentage points of gain. This further demonstrates that pre-execution selection cannot conjure value out of thin air in the absence of genuine specialization.

Even more telling: both groups (generated programs and baseline copies) reached 98.70% oracle coverage after 27 harness executions. The identical endpoint shows that specialization left no unique imprint on coverage.

Answer extraction is the final step in harness execution, responsible for identifying and extracting the final answer from the model's raw output — for example, parsing "so x = 42" as the number 42. This step seems simple but significantly impacts evaluation conclusions. Different extraction rules (e.g., how strict the regex is, compatibility with LaTeX formatting) can cause the same model output to be judged correct or incorrect. The "1 persistently winning task" in this paper was highly sensitive to the answer extraction method, suggesting its advantage may not stem from reasoning ability itself but from coincidentally matching a particular output format convention. This signals to researchers: when comparing different harnesses, the answer extraction logic must be kept consistent, otherwise format-matching differences will contaminate measurements of specialized capability.

Pinpointing Where Failures Occur

Using BIRD traces, the research localized specific failure points to three main areas: mechanism implementation, activation, and output validity.

This diagnosis is valuable. It suggests that problems with generated harnesses often aren't about having the wrong strategic direction, but about engineering-level failures in actually instantiating the mechanism, triggering it correctly, and producing output that meets specification. In other words, many failures are fixable implementation defects rather than fundamental capability gaps.

BIRD (a behavior-informed reasoning diagnosis mechanism or similar execution tracing tool) is a diagnostic approach that decomposes program execution into fine-grained steps to identify failure causes. Mechanism implementation failures indicate errors in the core algorithmic logic — such as an off-by-one error in a loop or a miscalculated formula. Activation failures mean the program is correctly implemented but fails to trigger under the right conditions — for example, a broken conditional branch causes a critical code path to be skipped. Output validity failures occur when the program executes successfully but produces output whose format doesn't meet downstream answer extraction requirements. Localizing failures to these three levels matters because they correspond to different remediation strategies: implementation errors require fixing the algorithm, activation errors require adjusting control flow, and output validity errors only require standardizing the output format. This layered diagnosis provides concrete engineering handles for improving the quality of automatically generated harnesses.

Setting a Harder Standard for Specialization

The core contribution of this work is demonstrating why coverage and repeatability alone cannot prove useful specialization. Stable scores and higher hit rates look attractive, but they may simply be byproducts of repeated execution.

Based on this, the authors propose an evaluation standard for harness diversity. Genuinely useful task advantages must satisfy three conditions:

• Persist across executions: the advantage cannot be a one-time stroke of luck; it must reproduce stably across multiple repetitions;
• Guide actionable decisions: the advantage must be exploitable by a selection mechanism and translate into real gains;
• Outperform additional fixed-program executions under a matched reasoning budget: the specialized program must be able to beat the naive baseline of simply running the base program more times.

The third criterion is especially critical — it puts "more programs" and "more repetitions" in direct competition under the same compute budget. If a carefully generated harness ensemble delivers gains no better than spending the same compute on repeated baseline runs, then the so-called specialization is hollow.

Implications for Practitioners

For teams exploring LLM agents, multi-program sampling, or automated reasoning frameworks, this paper offers a pragmatic warning: don't be seduced by coverage numbers. Before reporting specialization gains, ask yourself one question — could simple repeated execution with the same compute budget achieve a comparable result?

The paper also acknowledges that the problem of stable complementarity remains unresolved under three repetitions. This leaves room for future work: more repetitions, stricter budget matching, and attention to answer extraction robustness will be needed to truly clarify the value boundaries of harness diversification.

This kind of rigorous "demystification" research is especially important for avoiding self-deception in evaluation. When the entire field is chasing increasingly complex automated frameworks, an evaluation methodology that cleanly separates random gains from genuine capability is as valuable as any new method itself.