LeanZero
A team that develops agentic coding benchmarks and an evaluation platform, and that releases the Gauntlet and Forge series of benchmarks along with public datasets.
Timeline (last 90 days)
The LeanZero team packaged all row data, per-round build scores, inference costs, and replay links from its agentic coding benchmark into a public dataset released on Hugging Face, and built a leaderboard Space.
This benchmark is not a SWE-bench- or Terminal-Bench-style task but long-cycle agentic building of real applications, and the scoring checks were written by the team itself and may have blind spots.
In LeanZero's two agentic coding benchmarks, Haiku 5.5 won both rounds, with a combined billing of $1.62 for the two runs versus Luna's $0.46, a difference of approximately 3.6 times.
The author acknowledges the bug is a defect in their own fork of goose and has submitted a fix, estimating that with correct truncation, the cost for that run would have dropped from approximately $1.15 to approximately $0.48.
Each model was run only once on each leaderboard; a single run cannot distinguish model capability from luck, and all conclusions should be read as 'what happened in these runs' rather than as laws about the models.
All Facts (5)
This benchmark is not a SWE-bench- or Terminal-Bench-style task but long-cycle agentic building of real applications, and the scoring checks were written by the team itself and may have blind spots.
50%UnverifiedIn LeanZero's two agentic coding benchmarks, Haiku 5.5 won both rounds, with a combined billing of $1.62 for the two runs versus Luna's $0.46, a difference of approximately 3.6 times.
50%UnverifiedThe author acknowledges the bug is a defect in their own fork of goose and has submitted a fix, estimating that with correct truncation, the cost for that run would have dropped from approximately $1.15 to approximately $0.48.
50%UnverifiedEach model was run only once on each leaderboard; a single run cannot distinguish model capability from luck, and all conclusions should be read as 'what happened in these runs' rather than as laws about the models.
50%UnverifiedThe LeanZero team packaged all row data, per-round build scores, inference costs, and replay links from its agentic coding benchmark into a public dataset released on Hugging Face, and built a leaderboard Space.
50%