Agentic Coding Benchmark Goes Open Source: A Deep Dive into the Hugging Face Dataset and Leaderboard

When model leaderboards increasingly resemble marketing material, dumping the raw data — along with every single build record — onto Hugging Face is itself a move worth paying attention to. The LeanZero team recently did exactly that: they packaged the full row-by-row data from their agentic coding benchmark, per-round build scores, inference costs, and replay links into a public dataset, and built a small leaderboard Space on top of it.

What's in the Dataset

The dataset currently contains 94 rows, each corresponding to a model's performance on one leaderboard, covering three leaderboards in total. Gauntlet 7.2 requires models to build a payment app; Forge 1.0 and Forge 2.0 require building or upgrading an Atlassian Forge app for Jira.

Beyond the total score, each row preserves per-round build scores, recorded inference costs, and a link back to leanzero.net — where you can view check items, screenshots, and for some records, video. The data syncs automatically from the website every three hours, and the sorting logic reuses the website's own code, so the Space and the official site display consistent rankings. The authors mention that a check one morning found the dataset lagging one deployment version behind the site; they later fixed the sync logic to validate the entire leaderboard.

This means one thing: you can pull the data directly, sort it yourself, chart it yourself, and even challenge their conclusions. For researchers long frustrated by black-box leaderboards, this reproducibility is more valuable than the rankings themselves.

The Controversy at the Top: Jev Router vs. GPT-6.1 Sol

The most interesting part of the Forge 2.0 leaderboard is at the top. Jev Router ranks first with a median of 0.746, and OpenRouter's call logs show that 117 of its 118 calls went to GPT-6.1 Sol. When Sol competes on its own, it ranks third with a score of 0.5933. The same model did almost all the work, yet the result is better.

But the authors themselves pour cold water on it: this is not a clean comparison. Jev and Sol are in different comparison groups on the leaderboard; Sol's builds ran five days earlier, used an older goose build, and the router also selects reasoning effort per call. The author's own words: he wouldn't claim the results from the two rounds of builds are settled.

This detail is worth reading closely. It reveals an easily overlooked variable in agentic coding evaluation: routing and orchestration strategy itself may matter as much as the underlying model. The same model, called a different way, can jump from 0.59 to 0.75. Of course, confounding factors do exist, so it can't be attributed simply.

What Is a Model Router

The routing layer sits between user requests and the underlying models, dispatching each call to different models or different reasoning effort levels based on prompt type, context length, task complexity, or preset policies. Aggregation platforms like OpenRouter provide routing capabilities themselves, while Jev Router is a contestant that explicitly introduces a routing strategy into the benchmark. The value of routing lies in this: a single model may not be optimal for every subtask, and dispatching code generation, debugging, and tool calls to different backends can theoretically raise the overall ceiling. But it also creates an evaluation problem — when a single submission involves multi-model collaboration, it's hard to disentangle whether the score should be credited to the routing strategy or to a particular underlying model. This is exactly why the Jev Router vs. GPT-6.1 Sol comparison in the article isn't clean.

Pitfalls You Must Know Before Reading the Data

The authors candidly list the data's limitations — a section not commonly seen in leaderboard-style articles.

Each row for Gauntlet and Forge 1.0 has only a single run, and run-to-run noise is real. The authors explicitly state that if two models differ by only a few percentage points, it's not worth over-interpreting. Each Forge 2.0 row is the median of two independent builds, but the two can differ wildly — gemini-4-argon scored 0.7926 one time and only 0.2943 the other.

The cost fields are also incomplete. Gauntlet currently has only one row with recorded cost; Forge 1.0's figures are lower bounds because billing records are incomplete; most Forge 2.0 figures are complete. These caveats help readers judge credibility more than the rankings themselves.

Median vs. Single Run in Benchmark Evaluation

Agentic build tasks are highly stochastic: model sampling temperature, tool-call order, and transient environment states all cause the same model to produce different results across two runs. A score from a single run is a point estimate that cannot distinguish true model capability from incidental fluctuation; taking the median of multiple runs suppresses the influence of extreme values, but it's still only meaningful if dispersion is reported. In the article, gemini-4-argon's scores dropping from 0.7926 to 0.2943 is a classic case of huge variance — in such cases the median only describes "typical performance" and masks the risk of "sometimes very bad." Judging whether two models truly differ usually requires multiple repeated runs and statistical testing, not comparing a single number.

This Is Not SWE-bench

The authors specifically emphasize that these tasks are not SWE-bench or Terminal-Bench style tasks, but long-horizon agentic builds of real applications, with scoring check items written by the team themselves — and therefore inevitably carrying their own blind spots.

This positioning determines the dataset's applicable boundaries. It measures a model's overall performance on long-chain, multi-step engineering tasks with real deliverables, not single-point code-fix capability. For teams wanting to understand how agents perform in real development workflows, this kind of data is more informative than pure function-level benchmarks; but one must also recognize that self-built check items mean the scoring criteria themselves may introduce bias.

The Evaluation Paradigms of SWE-bench and Terminal-Bench

SWE-bench draws GitHub issues from real open-source repositories, requiring models to generate patches that pass the repository's existing tests. It evaluates single-point bug-fix capability, with clear task boundaries and automatic judging. Terminal-Bench places models in a terminal environment to complete a series of command-line tasks, emphasizing tool use and environment interaction. What both share: short tasks, objective judging, and large-scale automatic reproducibility. The benchmark in this article takes a different path: letting agents build or upgrade a runnable real application from scratch over a long horizon, with scoring relying on the team's self-built check items plus screenshot/video replays. This is closer to the full picture of engineering delivery, but sacrifices objectivity and reproducibility of judging, so it cannot be directly compared with SWE-bench scores.

How to Use This Data

If you're familiar with Python, a few lines of code can pull out the top ten of Forge 2.0. The dataset provides a leaderboard file in JSONL format; read it in with pandas, filter by version, and sort by rank — build scores can be rounded item by item and expanded for viewing. For anyone wanting to reproduce or do secondary analysis, this is the most direct entry point.

The authors close with an open question: if you run agents in your daily real work, does this leaderboard's order match what you see? Is there a model you feel is ranked clearly too high or too low? This posture of inviting challenge, combined with downloadable raw data, forms a relatively healthy closed loop for the evaluation ecosystem.

A leaderboard's value was never only in the rankings, but in whether it can be falsified. Publishing the data and evidence together at least makes falsification possible.