0.42 Points Apart on the Same Task: Breaking Down the DeepSeek V4.1 Flash 1017 Benchmark

A model launched just one day ago, called a "minor update," ran the same benchmark task twice and scored 0.2422 and 0.6618 respectively. That 0.42 gap is larger than the gap between almost any two models on that day.

According to a benchmark report from leanzero.net, DeepSeek^Ad released V4.1 Flash 1017 on Saturday, October 17. The company called it a minor update, with the API name and pricing unchanged—calling deepseek-flash automatically points to the latest model. It beat the September version on both agentic coding benchmarks: 0.6957 vs. 0.591 on Gauntlet 7.2, and a 0.4520 vs. 0.3342 median on Forge 2.0. But the real story isn't these two numbers—it's the huge variance between two builds of the same model on the same task, and the string of issues the evaluators exposed in their own setup.

Two Benchmarks and One Caveat

The evaluation uses two benchmarks. Gauntlet (sb-7.2) has goose build a small payment app with two services and a 3D view, giving the model 150 calls, then runs 99 checks. Forge 2.0 hands goose a working Jira Forge app and a long v2 spec, gives it 300 calls, then runs 105 checks against three pre-provisioned Jira sites. A single model call can request multiple tool calls, which is why Nemotron racked up 397 tool calls within a 300-call budget. The Forge 2.0 leaderboard lists each model as a pair of independent builds; the Gauntlet leaderboard shows single runs.

The authors proactively concede one point up front: the numbers come from one task per benchmark and one build or one pair of builds per model, and their evaluation environment (an unreleased goose build, the Goose Swarm desktop app, a custom MLX engine) changed between September and October. As a result, DeepSeek's two Forge pairs fall into different comparison groups, and part of the "beats the previous generation" conclusion may come from the evaluators themselves.

Agentic coding benchmarks refer to a class of evaluation tasks where the model doesn't answer a single question, but instead works inside a real code repository, making multiple rounds of tool calls (reading and writing files, running shell commands, executing builds, launching a browser to self-test), independently completing a runnable software project, which is then scored item by item by automated check scripts. The key difference from traditional code benchmarks (like function completion in HumanEval) is that the score depends not only on whether a single piece of code the model wrote is correct, but also on whether it can plan long tasks, self-correct from error feedback, manage its context budget, and whether it will "consider itself done" when no one is supervising. As a result, these benchmarks typically give the model a fixed call limit (e.g., 150 for Gauntlet, 300 for Forge) and run dozens to hundreds of functional checks against the final app. This also explains why two builds of the same model can show huge variance—the longer the task chain, the more any single step's path choice gets amplified by subsequent steps.

The First Build Got Stuck in an Inherited Retry Loop

Both Forge 2.0 builds used the same task fingerprint, the same starting app, and the same three scoring sites. Build 1 made 262 tool calls in 28 minutes and scored 0.2422; Build 2 made 270 in 30 minutes and scored 0.6618. Work was 0.8362 vs. 0.9693—not a big gap; the reliability multiplier was 0.2896 vs. 0.6828—that's where the gap is. Build 1 failed 15 of 19 check tiers; Build 2 failed only 9.

The most obvious failure came from a popup: on one of the scoring sites, Build 1's sprint popup opened 12 times, and all 12 were blank—no console errors, no failed requests. The scorer's observation file shows the parser returned results like {"rateLimited": true, "retryAfter": 2237} 14 times, and Build 1's static/shared/bridge.js retained the starting app's retry loop—it dutifully waited 2,237 seconds (about 37 minutes) before retrying, drawing nothing on screen the whole time. Build 1 rewrote this file at 16:50 but kept the loop verbatim. Build 2 called invoke directly and rendered a message on failure.

Why did Build 1 trigger rate limiting? The authors' interpretation (not proof) is that the task's rate model gives the app a 2,400-point hourly quota, with background work allowed up to 1,680 points. Build 1 allocated up to 1,350 points of independent budget to each scheduled reconciliation job and wrote down "Worst = 1350+180+100 = 1630 < 1680," an equation that assumes reconciliation runs once per hour. The scorer actually launched it twice, 8.5 seconds apart, consuming 787 and 732 points respectively; that hour's quota was exhausted and requests from 25 representative users were rejected. Build 2 re-read the shared counter before each request, consuming 1010 and 175 points across the two runs. The authors also admit they can't explain how Build 2's own hour reached 2,436 points with no one rejected.

Build 1 actually saw the wall in its own testing: at 16:52 it wrote "Quota exhausted!? Retry-After 938 s."; at 16:59 it concluded other tenants had used up the quota, reset the dev site, and continued—never testing what the UI would do when the parser returned "rate limited." Its closing line was "The app is complete and validated."

The authors explicitly don't sing Build 2's praises: it removed the loop for its own reasons (only one call allowed before first paint), not out of foresight about quota issues; it never hit a 429 during scoring, and its error path was equally untested. Part of the 0.42 gap is luck about where the pressure landed.

Exponential backoff and Retry-After semantics are key background for understanding Build 1's failure. An HTTP 429 (Too Many Requests) response typically carries a Retry-After field telling the client how many seconds to wait before retrying; a value like 2237 returned by the server is meant to make the client delay retries to relieve pressure. A reasonable client implementation would set a cap (e.g., wait at most tens of seconds) and show the user a loading or error state while waiting; but the loop in the starting app had neither a cap nor any UI feedback, so when the model carried it over, the result was "popup opened 12 times, all 12 blank." The 1680/2400-point quota model mentioned in the report is another common design: an hourly token bucket where background tasks share the same counter as user requests; once a reconciliation job reserves too large an amount based on worst-case assumptions, it can eat up the whole hour's headroom at once, causing subsequent legitimate requests to be rejected. Build 1's calculation error was precisely treating "runs once per hour" as a given.

Nemotron 3.5 Super VL: Wins on Speed First, Loses on Context

NVIDIA's Nemotron 3.5 Super VL is a 120B-total-parameter, 12B-active Mamba-2–Transformer hybrid MoE supporting up to 1M context, priced on OpenRouter at $0.10 per million input tokens and $0.40 per million output tokens, served by five hosts, four of which only serve a 262,144-token window.

On Gauntlet it wrote its first file in 39 seconds, writing 1,354 lines of backend in one go. When goose's edit tool rejected its edit calls seven times in a row due to parameter issues, it switched to Python string replacement in the shell (27 edits) to keep going, and ran 19 browser self-tests. 41 of 99 checks passed completely, and the score was capped at 0.599 because its 3D panel rendered as an empty canvas.

The problem is that after a self-test report of "Everything works with zero errors," it said it wanted to look at a screenshot to verify the tower, but no Python image library was installed on the machine; the fallback it wrote printed only the PNG file header and the file size of 398,867 bytes, and then the model moved on—without looking at a single pixel. DeepSeek 1017 on the same day used goose's image tool to open seven of its own screenshots, and its 3D view passed the visible-surface check. Nemotron is also a vision model with the same tools, but it chose Python.

Build 1 on Forge 2.0 split into three phases: 8.6 minutes reading the contract, starter code, offline Forge toolkit, and manifest schema; about 7 minutes completing all 15 writes and 4 edits; then 290 tool calls and 11 minutes with no writes, lint, or build. It requested the contract another 81 times (39 of which returned content), the starter code 70 times, the rate model 42 times, and 68 of the 290 calls were aimed at a nonexistent read tool.

At second 931 it was still saying "Now let me write the explain.js…"; at second 975 it was like it was on its first day: "Let me start by reading the contract and starter files to understand what's required." The run ended with "Let me read the remaining source files."—after which there were no tool calls and no summary.

The prompt token count recorded by its harness dropped abruptly from 146,368 to 7,394 and never exceeded 67,997 again. goose didn't discard history (the session database is fully preserved, with the last request log showing 824 messages and 5.5 million characters sent), but OpenRouter's middle-out transform was enabled. According to OpenRouter's docs, middle-out deletes or truncates messages from the middle of the prompt until it fits the context window, keeping the oldest and newest messages. Nemotron's behavior matches this exactly, but the authors can't prove it actually happened.

The authors also acknowledge problems with their own setup: Nemotron wasn't pinned to a host, and goose took the 1,048,576 context ceiling from the model list, while DeepInfra—which actually served 225 of the 229 calls—accepts only 262,144; goose added middle-out to all OpenRouter requests, so goose itself never compacted, but the downstream did the trimming.

The direct cause of 0.0321 was an edit made before the truncation: adding getSprint to the import on line 5 of src/views.js, while line 68 of the same file defined a function with the same name. Every feature in the app imports this file, so all nine handlers failed to bundle with "The symbol "getSprint" has already been declared." DeepSeek 1017 in Build 2 carried the full history to 372,546 prompt tokens, and was the one of the two builds that bundled successfully.

Local Nemotron: A Tool Format the Parser Doesn't Recognize

On the local side, the evaluators tried Nemotron via the MLX engine on a MacBook with more memory, using 69.5 GB of 4-bit weights. That day it failed to build anything because goose couldn't talk to it: with tools, model calls returned as raw text <tool_call><function=write>..., and the engine parsed zero of them.

The background: the model doesn't "call tools"—it writes out the format text it learned during training, and the inference server must convert that text into a structured tool_calls field—that's the tool parser; another reasoning parser separates thinking from the answer. The evaluators' engine identified parsers through three layers: known aliases, a name-pattern table, checkpoint metadata, and chat templates—Nemotron matched none of the three. Its chat template only showed the tool format as examples in instructions, so the detector missed it; without a profile there was no reasoning parser either, and Nemotron's closing tag </think> was a single token (id 13) but wasn't marked as special in the tokenizer, so it was stripped during streaming and the answer became "no visible reply" in goose's eyes.

The fix is on the fork branch lz/nemotron-h-metadata: map the nemotron_h architecture to the nemotron tool parser and Qwen3-style reasoning, mark it as hybrid, and mark it as MoE when routing experts. Two new tests fail before the fix and pass after. After the fix, the 120B model cleanly answers "OK" on a Mac Studio, parses write calls 3/3 with and without reasoning effort, and separates 1,030 to 2,142 characters of reasoning content from the answer. For comparison, NVIDIA's model card suggests vLLM users manually pass --tool-call-parser qwen3_coder --reasoning-parser nemotron_v3. The fix shipped with Goose Swarm 3.0.119 on October 19.

The tool-call parser and the reasoning parser are two often-overlooked components in the local inference stack. The model weights themselves only output text tokens; so-called "tool calling" is the model writing text like <tool_call><function=write>... according to templates it saw during training, which the inference server then converts into the OpenAI-compatible tool_calls structure via regex or grammar parsing so the goose client can recognize it. The reasoning parser is responsible for separating the thinking content inside ... from the final answer, preventing chain-of-thought from leaking into the user-visible reply. Different model families use different tags, aliases, and chat templates, so servers like vLLM and SGLang require users to explicitly specify parser names; once an architecture isn't mapped, the server silently treats it as plain text, manifesting as "the model has output but no tools are called." The fork fix in the report is essentially filling in this metadata mapping layer and marking closing tags like <｜end▁of▁thinking｜> as special tokens so they aren't stripped as ordinary text by the tokenizer during streaming.

Muse Glimmer 30B: Clean Lint, Backend Never Bundled

Muse Glimmer 30B is a 29.6B dense model under Apache 2.0, with one global attention layer for every three sliding-window attention layers. It ran at 8-bit on a 64 GB MacBook.

It started well: it read the entire starting code before acting, and in its first reasoning pointed out a real ambiguity between two sections of the contract. Then at 20:04 it appended an admin resolver to src/index.js—line 182 const adminResolver = new Resolver();, line 186 also export const adminResolver = adminResolver.getDefinitions();, declaring the same name twice. Forty-four lines above, the starting code showed the correct pattern: the local variable was resolver, exported as uiResolver. Glimmer copied the layout but kept only one name.

It never touched that file again. Two minutes later goose compacted the context (105k down to 51k tokens), and the remaining 27 minutes went almost entirely to the manifest: 11 more lint runs. The final npm run build built only the three frontend bundles, and it wrote a summary: lint clean, all v2 modules declared, admin resolver enforces ADMINISTER on the server. The authors' assessment: three of the four sentences are true, and the key one is false—lint was clean only because lint reads only the manifest; "bundles and loads" wasn't verified by anything it ran. Under the scorer, all 8 functions failed to bundle at line 186, including the v1 functions it never touched; 0 of 22 resolver calls returned results; score 0.0325.

The authors also flag a trap: trying to catch this error with node --check won't work—the workspace has no "type": "module", and under Node 24 node --check reports nothing for a .js file using export. What actually catches it is the forge-dev invoke named in the starting code, or any esbuild backend bundle.

ESM and the node --check blind spot deserve a separate note. JavaScript's module syntax depends on how the file is interpreted: with "type": "module" declared in package.json, .js files are parsed as ES Modules and export is legal; otherwise they're parsed as CommonJS, where export should be an error. But node --check only does syntax checking, not module resolution, so under Node 24 it may report nothing for a .js file using export without a declared type—so "syntax check passed" gives models and developers a false sense of security. What actually exposes the duplicate const declaration is a backend bundler (like esbuild) doing binding checks while resolving the module graph, or a command like forge-dev invoke provided by the Forge runtime that actually loads the app entry point. This is also the technical root of the report's emphasis that "clean lint doesn't mean the app runs": lint tools usually only read declaration files like the manifest and don't build the module dependency graph, so they're completely blind to cross-file symbol conflicts.

Mistral Large 4: One Failure Blamed on the Evaluators Themselves

Mistral Large 4 is a 1.05T-total-parameter, 52B-active MoE, in Public Preview since October 6, with only Mistral as a host on OpenRouter. Its October 9 Gauntlet attempt ended after 49 calls due to an upstream idle timeout; in the October 18 run, it thought first: a single thinking call of about 38,000 words ran for 18 minutes, with a repetition rate below 1% for 12-word sequences—coherent planning rather than a loop. The first source file appeared about 32 minutes in, then requests began failing; at 19:40 the Mistral endpoint returned 503 "Internal server error," goose resent three times with the same result, and the run ended after 46 minutes and at least $1.18.

The next attempt's failure counts as the evaluators' own: when a tool result carries an image, goose appends a pure-image user message, and with parallel tool calls this message lands between two tool results; Mistral's API is strict about message order and returned "400 Unexpected role tool after role user," ending the run after 97/150 calls. The issue was fixed in Goose Swarm 3.0.119 (image messages moved after the last tool result), and the fourth attempt on October 19 completed: 144 minutes, $4.93, 0.5462. Before the critical-defect multiplier it was 0.9104; one check deducted 40% of the score, and that one was also the evaluators' fault: the on-screen amount check compared against the oldest 50 payments, while Mistral's table showed the newest by default, and all 24 amounts the probe read were correct.

Billing: The Money the Leaderboard Didn't Show

Until Sunday, Gauntlet runs had no price on the leaderboard, and most Forge runs showed only "at least" figures. The evaluators went back to OpenRouter's generation records to reconstruct every bill: 36 Gauntlet runs got prices for the first time (17 complete, 19 marked "at least"), 33 Forge bills improved, 49 runs remain Not recorded, and 94 rows that previously showed raw model ids were renamed.

The numbers themselves tell a story: a pair of DeepSeek 1017 builds cost $0.62, a fraction of the most expensive Forge 2.0 combos (GLM 5.3 Prime at $99.66, Qwen3.8 Max Prime at $72.04). Mistral Large 4's Forge pair cost at least $5.40 plus $5.55—more than seventeen times DeepSeek 1017—with a lower median.

Three Things Worth Remembering

First, a single run isn't enough to judge an agentic model. DeepSeek 1017's two builds on the same task and seed differed by 0.42, and the clearest difference is a rate-limit retry loop that one inherited from the starting app and the other dropped.

Second, when a long task stops writing code, first confirm whether the prompt your provider sees has been shortened before blaming the model. Nemotron's 290 tool calls with no output happened after a prompt dropped abruptly from 146k to 7.4k.

Third, passing lint doesn't mean the app runs. Muse Glimmer's manifest passed cleanly in its last two runs, yet none of the 8 backend functions could bundle; what actually catches this is the Forge runtime's forge-dev invoke or a backend bundle check.

According to the report, these numbers come from one task per benchmark and one build or one pair of builds per model, and the evaluators have publicly listed several of their own mistakes, including the leaderboard missing costs, Nemotron not being pinned to a host, and uniformly requesting medium reasoning effort for all cloud models while each provider's actual value differed. For readers who want to reproduce or cite these results, these caveats matter as much as the scores themselves.