Gauntlet
A benchmark for agentic coding that requires a model to build a runnable application from scratch within a limited number of calls, with each item scored by automated check scripts.
Timeline (last 90 days)
The Gauntlet (sb-7.2) benchmark has goose build a small payments app with two services and a 3D view, gives the model 150 calls, and then runs 99 checks.
On the Gauntlet, Nemotron 3.5 Super VL fully passes 41 of 99 checks, with its score capped at 0.599 because its 3D panel rendered as an empty canvas.
All Facts (2)
The Gauntlet (sb-7.2) benchmark has goose build a small payments app with two services and a 3D view, gives the model 150 calls, and then runs 99 checks.
50%UnverifiedOn the Gauntlet, Nemotron 3.5 Super VL fully passes 41 of 99 checks, with its score capped at 0.599 because its 3D panel rendered as an empty canvas.
50%