The Sparse-Reward Excuse Is a Cop-Out: The Real Bottleneck in Agent Evaluation Is Authoring Cost

The Sparse-Reward Excuse Is a Cop-Out: The Real Bottleneck in Agent Evaluation Is Authoring Cost

Long-horizon agent evaluation keeps getting stuck in the same place: you want to test a task that runs 40 to 100 steps, but you can't score it densely. Step-by-step scoring means a human or a judge model has to read the entire trajectory — expensive, slow, and noisy. So everyone settles for the next best thing: checking only the final assertion, which gives you a black-or-white pass or fail.

That kind of evaluation is cheap, but it's nearly useless. When the agent fails at step 47, all you have is a terminal error and a full conversation log, and you spend the whole afternoon playing detective, going through it line by line. The signal you actually need appeared back at step 12, but you didn't collect it, because collecting it looked too expensive.

The root of the problem is this: dense scoring was never the expensive part — authoring is. We've been blaming the wrong cost all along.

What Dense Scoring Actually Requires

To say "progress was made at step 14," something has to know what "progress" at step 14 means. If all you have is a task prompt and a final assertion, nothing knows that definition. So you hand the trajectory to a judge model and let it guess. That's expensive, and it's guessing, because the judge model has no real standard either. It reads the same record you do, lacks the same information you lack, and charges you by the token.

Tev took a different path. Instead of post-hoc trajectory scoring, it first authors a hidden ground-truth world specification — the world's actual state, preconditions, milestones, termination conditions — and then renders the environment from that spec. The environment isn't the kind you hand-build and then write docs to describe, with the docs drifting over time; the spec is the source, and the environment is just its projection. That way, progress at any step is a function of the spec — no judgment required, just computation. Dense per-step reward emerges naturally as a byproduct, because you've already written down the definition of "closer."

The specifics are in the paper: arxiv.org/abs/2610.18868v1

Sparse vs. dense reward is a contrasting pair of concepts in reinforcement learning. Sparse reward gives a signal only at the end of the task — success or failure — with no feedback at all in between. Dense reward gives a progress measure at every step or milestone. Sparse reward's advantage is low authoring cost and less susceptibility to human bias; its drawback is that credit assignment is hard — the agent struggles to know which step in a long action sequence caused the failure, and the learning signal is extremely thin. Dense reward is the opposite: it provides a smooth learning gradient, but only if there exists an objective definition of "progress." Otherwise every step's feedback is a subjective judgment — neither reproducible nor scalable.

World specification refers to a machine-readable description of the environment a task lives in: the system's initial state, legal state space, invariants that must always hold, milestones that define stage completion, and the conditions under which the task terminates. Its key difference from the traditional "task prompt + final assertion" is explicitness — the prompt is just a natural-language description of human intent, and the assertion covers only the endgame; the spec writes "what counts as right" for the whole process into code. When the spec becomes the single source of truth, the environment can be rendered from the spec, the observed state at any moment can be compared against it, and progress therefore becomes a computable function rather than a judgment a judge model has to guess at.

Why This Matters Just as Much in Production

What really made me care about this idea wasn't the benchmark numbers — it's that it happens to be exactly the discipline we've never managed to implement in production. Long-horizon agents call tools, hold state, and run continuously for minutes at a time. When they go wrong, my dashboard only tells me something went wrong; it doesn't tell me the agent started drifting at step 9 and was unrecoverable by step 20. I have logs, I have traces, but I have no "expected state" to compare against, so all that information is only descriptive, not diagnostic. I can state exactly what the agent did, but I can't say clearly from which step it went off track.

The solution isn't to find a smarter judge model to watch the agent in real time. I tried that. It's slow, it charges by the step, and in exactly the ambiguous middle ground where I most need it to be right — when the agent is doing something that looks plausible but deviates from the spec — it confidently gives the wrong verdict. The real solution is to write down the world specification: what the state should look like, which invariants should hold, what the milestones look like. Then compare observed state against it. That check is cheap — it's comparison, not reasoning. It gives you a progress curve, not a post-mortem report.

Instrumentation contract is the product-side extension of this idea. Traditional observability only records events — logs, traces, metrics — and answers "what did the agent do." A contract additionally specifies "what the expected state should be," checking actual state against it after every tool call, thereby upgrading descriptive data into diagnostic data and answering "from which step did it go off track." This practice is nothing new in distributed systems: health checks, assertions, and SLOs are all essentially comparisons of expected state against observed state. The difference is that an agent's intermediate state is far harder to characterize than a service process, which makes an explicitly written world specification all the more necessary as a comparison baseline.

Myths to Bust and Four Principles Worth Borrowing

The myth I want to bust is this: dense agent evaluation is too expensive, you can't afford to run it. Not so. It's too expensive to improvise. Without a spec, every step is a subjective judgment, and subjective judgments can't scale — not in money and not in consistency. With a spec, every step is a table lookup.

Here are four things worth borrowing specifically from Tev:

1. Write the world specification first, then the task prompt. State, invariants, milestones, termination conditions. If you can't write it, your understanding of the task isn't yet sufficient to evaluate it, and no number of judge models will fill the gap.

2. Render the environment from the spec. Single source of truth. A hand-built environment drifts from its description, and then your reward is scoring a world that doesn't exist.

3. Let progress emerge as a byproduct, not as a second pass. If scoring is a separate pipeline, it rots the moment the environment changes. If it's a projection of the spec, it stays in sync automatically.

4. In production, use that same spec as your instrumentation contract. Check expected state after every tool call. What you get is a curve, not a feeling.

Where to Be Wary

How well rendering-from-spec works depends on the quality of the spec itself. If the spec isn't specific enough, you get confident but wrong dense rewards, and an agent that optimizes straight into the gap between your spec and reality. That's worse than sparse, because sparse failure is at least honest — it doesn't tell you you're winning.

Also, writing specs is real work. You're moving the cost, not eliminating it. But moving it from per-run scoring to one-time authoring is the right trade-off, because specs are reusable, reviewable, and versionable. A wrong spec is a bug you can fix; a noisy judge is a fog you can't dispel.

Dense reward also brings debuggability. Sparse gives you pass or fail plus a feeling; dense gives you a curve, and a curve tells you where the agent stalled. That's the real product — not the score, the shape.

Next time someone tells you long-horizon agent evaluation has to be sparse because dense scoring is too expensive, ask them for the world specification. If they don't have one, that's the cost they're really paying — they just aren't listing it.