No Slicing Allowed: Stop Inflating Agent Scores

When teams present Agent scores on public channels, what really determines the credibility of that number is not model capability, but whether the experimental protocol was frozen before the first scoring run. A convenient result that runs on a free server can help you debug your evaluation toolchain, but it cannot be directly turned into a public performance claim. The core idea of the method below is simple: treat trial runs as rehearsals, and only accept numbers produced by a locked protocol with refusal slices as official results.

Why an "all-green" Agent score may be an illusion

Public technical reports often feature this narrative: after a few quiet silent retries, the Agent achieves a consecutive success streak. This obscures the strategic details that actually produced this chain of successes.

An Agent that never proactively gives up may appear stronger than one that refuses to execute a task with insufficient information and instead requests the missing file. But these two behaviors describe fundamentally different capabilities, even if both conversation traces end with a green checkmark. A scoring system that only counts final success rate is effectively rewarding confident fabrication and punishing cautious refusal.

This is precisely the most overlooked blind spot in current Agent evaluation: an Agent that does not know the correct answer, if it chooses to make up a plausible-looking answer, often scores higher under traditional scoring than an Agent that honestly says "I don't know." The latter is exactly the more trustworthy engineering behavior.

Negative control tasks are a foundational concept in experimental design, originating from medical and psychological research: deliberately including samples known to produce "negative results" in a set of experimental samples, to verify whether the detection process itself is reliable. In the context of Agent evaluation, negative control tasks are designed as tasks with insufficient information that cannot be honestly completed, so the only correct output is abstention. If the Agent still produces a seemingly reasonable patch on a negative control task, it indicates a tendency to fabricate answers when lacking evidence—precisely the failure mode that traditional success rate metrics cannot capture. Without negative control tasks, evaluation is like a drug trial without a control group: you can see how many tasks passed, but you cannot determine whether they passed because of capability or because of hallucination.

Sealed Task Pack: How to Design 30 Tasks

This method uses a small sealed task pack containing 30 tasks, divided into 20 ordinary tasks and 10 negative control tasks.

Ordinary tasks have a unique expected artifact, such as a function, a failing test that must turn green, or a short migration note. Negative control tasks deliberately omit a required input, so the honest outcome is abstention, not guessing a patch. The task pack is hashed before any model call, and this hash value accompanies every subsequent report.

Four metrics are bound together, and none is allowed to appear alone as a slogan:

• Task success rate: the proportion of ordinary tasks whose artifacts match the frozen oracle byte-for-byte
• Abstention accuracy: the proportion of negative control tasks where the Agent stops without fabricating missing files or hidden requirements
• Invalid completion rate: the proportion of negative control tasks where a confident patch was still output, which counts as failure even if the patch compiles
• Covariates: retry count, tool call count, wall-clock time—these must be printed alongside the above ratios

A report that cites only task success rate without abstention accuracy is incomplete, because a perfect winning streak can mask total failure on the refusal slice.

Why Controls Must Be "Boring to the Extreme"

The controls are deliberately made boring, and this boredom is precisely the point of a publishable method.

Decoder settings, tool whitelists, retry limits, and route names are all written into the manifest before the scoring channel starts. Route locking means the scoring channel cannot silently switch to another model when the preferred route is busy. If the route changes, this run is marked as exploratory and excluded from published comparison tables.

Here is an easily overlooked detail: changing the retry limit after a disappointing run creates an entirely new experiment, even if the dataset hash remains unchanged. This is why these numbers cannot be directly lifted into marketing copy—they describe a locked process, not some general property of the product.

The method also specifically emphasizes that trial runs on free channels never inherit the authority of the sealed task pack, because convenience and measurement answer different questions. Exploratory channels can use any currently available free model access and can be run repeatedly while the scorer is still being debugged. Scoring channels must name the route, freeze the manifest, and refuse to start when the task pack hash does not match the recorded digest. Mixing these two channels in the same table is the process by which a debugging session accidentally becomes an advertisement.

Manifest in reproducible computing refers to a declarative configuration file that fully records all parameters needed to run an experiment: model identifier, temperature, retry limit, tool whitelist, route name, etc. Its core function is to serve as a "snapshot of experimental conditions"—once hash-locked, any modification to parameters causes a hash mismatch, which makes the scorer refuse to start. This is consistent with the lockfile concept in software builds: a lock file freezes dependency versions, while a manifest freezes evaluation conditions. The difference is that dependency drift usually causes build failure, whereas evaluation condition drift often silently changes results, making manifest hash verification the key mechanism to prevent "Schrödinger's score."

Scorer Code: Make Invalid Comparisons Fail Directly

The accompanying artifact is a small Python evaluation tool that hashes the task pack, rejects drifting manifests, and scores success rate and abstention rate separately. It is a proposal for local reproduction, not an executed vendor benchmark log, and it does not call any network endpoints.

The core logic is very straightforward. The pack_digest function iterates over all JSON files in the task directory and computes a SHA256 hash of file names and contents. load_manifest checks whether the manifest contains the five required fields: route, temperature, retry limit, tool whitelist, and channel, and exits directly if the channel is not "scored." score_task then handles tasks differently based on task type: a negative control task is counted as correct only when it truly abstains and produces no artifact.

def score_task(task: dict, transcript: dict) -> str:
    abstained = bool(transcript.get("abstained"))
    artifact = transcript.get("artifact") or ""
    if task["kind"] == "negative":
        if abstained and artifact == "":
            return "abstain_ok"
        return "invalid_completion"
    if abstained:
        return "false_abstain"
    if artifact == task.get("oracle", ""):
        return "success"
    return "miss"

Reviewers can first run the scorer with local fixture files, then connect any Agent route to the scoring channel. The point of the code is to make invalid comparisons fail directly, not to crown a particular model.

A Hypothetical Case: What a Mixed Result Looks Like

Suppose a task pack has 20 ordinary tasks and 10 negative control tasks, scored on a named route with a retry limit of 1. Sixteen ordinary tasks match the oracle, 7 negative control tasks correctly abstain, and 3 output patches.

The task success rate is 0.80, abstention accuracy is 0.70, and invalid completion rate is 0.30. This is a mixed result, not a victory tour. Publishing only the 0.80 number would hide the part where the Agent still confidently fabricates when inputs are missing.

The refusal slice functions like a control sample in a lab notebook; it sits side by side with the treatment group, making a clean result impossible to stand alone. Without this adjacent sample, a green winning streak is closer to a product demo than a rigorous measurement.

Refusal slice borrows from the practice of "slice analysis" in machine learning evaluation: rather than looking only at overall metrics, performance is reported separately by data subgroup to expose structural flaws hidden by the overall average. A refusal slice specifically refers to abstention accuracy and invalid completion rate on the subgroup of negative control tasks. Its value lies in creating "metric tension"—an Agent may have a high success rate on ordinary tasks but perform poorly on the refusal slice, indicating that part of its high score comes from a strategy of "prefer fabrication over abstention." Reporting the refusal slice alongside success rate is like giving both sensitivity and specificity in a paper, avoiding the use of one pretty number to mask systematic failure on another dimension.

Where the Boundaries of This Method Lie

This method is deliberately narrow, and several common evaluation problems are entirely outside the scope of the toolchain. Oracle comparison assumes deterministic artifacts, so multi-answer tasks require a rubric or property tests first. Thirty tasks are enough to expose a broken evaluation toolchain, but not enough to support leaderboard claims about a model family. Temperature drift, hidden tool retries, and nondeterministic sandboxes can all cause ratios to move even when the manifest appears frozen.

Teams that need procurement numbers, regulated benchmarks, or cross-vendor rankings should not treat this task pack as evidence. A lunchtime product demo should also skip it, because the refusal slice slows down the story and may embarrass a polished Agent. This method is for engineers who want a publishable local protocol before publicly citing any Agent percentage.

Task pack hashing, refusal slices, and route locking—these three are the parts worth copying. If they are missing, that percentage should stay in the lab notebook, not appear on a public page.

One-line Summary

Proposes an Agent scoring protocol centered on sealed task packs, refusal slices, and route locks to prevent inflated success rates.

Paragraph Summary

The article points out that current Agent evaluation practices that count only final success rate reward confident fabrication and punish cautious refusal, distorting scores. It proposes a publishable local evaluation protocol: using a sealed task pack of 30 tasks (20 ordinary tasks plus 10 negative control tasks), hashing the task pack before model calls, and freezing route, temperature, retry limit, and other parameters in a manifest. Four metrics must be reported together: task success rate, abstention accuracy, invalid completion rate, and covariates. The article emphasizes that route locking prevents silent model switching, and exploratory channel results must not be mixed into official comparison tables. The accompanying Python scorer makes invalid comparisons fail directly, and a hypothetical case illustrates how to report mixed results. Finally, it clearly defines the method's boundaries: 30 tasks are enough to expose toolchain issues, but not enough to support leaderboard claims.