Forge 2.0
A benchmark for evaluating the coding capability of agents, scoring models through repeated build tasks.
Timeline (last 90 days)
Forge 2.0 Build 1 made 262 tool calls in 28 minutes and scored 0.2422, while Build 2 made 270 in 30 minutes and scored 0.6618; Build 1 failed 15 of the 19 check tiers, while Build 2 failed only 9.
The Forge 2.0 benchmark hands goose a working Jira Forge app and a long v2 specification, gives 300 calls, and then runs 105 checks against three pre-provisioned Jira sites.
DeepSeek V4.1 Flash 1017 has a median of 0.4520 on the Forge 2.0 benchmark, exceeding the September version's 0.3342.
The same DeepSeek V4.1 Flash 1017 model scored 0.2422 and 0.6618 in two Forge 2.0 builds, with both builds using the same task fingerprint, the same starting application and the same three scoring sites.
Forge 2.0's rate model gives an application a quota of 2400 points per hour, with background work allowed to consume at most 1680 points.
In Forge 2.0 Build 1, the scoring site's sprint popup opened 12 times and was blank every time, the parser returned `{"rateLimited": true, "retryAfter": 2237}` 14 times, and Build 1's `static/shared/bridge.js` retained the starting application's retry loop.
The dataset's cost field is incomplete: only one row in Gauntlet records cost; the Forge 1.0 figures are lower bounds; most Forge 2.0 figures are complete.
The dataset currently contains 94 rows, each corresponding to a model's performance on one leaderboard, covering the three leaderboards Gauntlet 7.2, Forge 1.0, and Forge 2.0.
The Gauntlet 7.2 benchmark requires models to build a payments app; Forge 1.0 and Forge 2.0 require building or upgrading an Atlassian Forge app for Jira.
When competing on its own, GPT-6.1 Sol ranked third on the Forge 2.0 leaderboard with a score of 0.5933.
4 more timeline events
All Facts (14)
Forge 2.0 Build 1 made 262 tool calls in 28 minutes and scored 0.2422, while Build 2 made 270 in 30 minutes and scored 0.6618; Build 1 failed 15 of the 19 check tiers, while Build 2 failed only 9.
50%UnverifiedThe Forge 2.0 benchmark hands goose a working Jira Forge app and a long v2 specification, gives 300 calls, and then runs 105 checks against three pre-provisioned Jira sites.
50%UnverifiedDeepSeek V4.1 Flash 1017 has a median of 0.4520 on the Forge 2.0 benchmark, exceeding the September version's 0.3342.
50%UnverifiedThe same DeepSeek V4.1 Flash 1017 model scored 0.2422 and 0.6618 in two Forge 2.0 builds, with both builds using the same task fingerprint, the same starting application and the same three scoring sites.
50%UnverifiedForge 2.0's rate model gives an application a quota of 2400 points per hour, with background work allowed to consume at most 1680 points.
50%UnverifiedIn Forge 2.0 Build 1, the scoring site's sprint popup opened 12 times and was blank every time, the parser returned `{"rateLimited": true, "retryAfter": 2237}` 14 times, and Build 1's `static/shared/bridge.js` retained the starting application's retry loop.
50%UnverifiedThe dataset's cost field is incomplete: only one row in Gauntlet records cost; the Forge 1.0 figures are lower bounds; most Forge 2.0 figures are complete.
50%UnverifiedThe dataset currently contains 94 rows, each corresponding to a model's performance on one leaderboard, covering the three leaderboards Gauntlet 7.2, Forge 1.0, and Forge 2.0.
50%UnverifiedThe Gauntlet 7.2 benchmark requires models to build a payments app; Forge 1.0 and Forge 2.0 require building or upgrading an Atlassian Forge app for Jira.
50%UnverifiedWhen competing on its own, GPT-6.1 Sol ranked third on the Forge 2.0 leaderboard with a score of 0.5933.
50%UnverifiedIn Forge 2.0, the median gap between two builds of a single model is 0.0832, Jev Router's two builds differ by 0.2125, and Gemini 4 Argon's two builds differ by 0.4983 (0.7926 vs. 0.2943).
50%UnverifiedForge 2.0 requires each model to run two independent builds and ranks by median, and the leaderboard requires comparing models within the same comparison group; Jev Router and Gemini 4 Argon belong to comparison group 1, while Claude Opus 5.5 and GPT-6.1 Sol belong to comparison group 2.
50%UnverifiedWhen GPT-6.1 Sol competed alone, its median across two builds was 0.5933, ranking third.
50%UnverifiedTypeSafe's Jev Router scored 0.6398 and 0.8523 on two builds on the Forge 2.0 agentic coding benchmark, with a median of 0.746, ranking first.
50%