Local Quantized Models vs Claude: 9 Real Bugs Tested, Who Comes Out Ahead?

When it comes to local models, community discussion tends to revolve around tok/s and quantization loss—but what really determines productivity is whether the model can actually fix the bugs in front of you. A developer on Reddit ran a comparison test using 9 real regression bugs from their own engine, putting local quantized models and the Claude family under the same scoring system. The results are far more interesting than raw benchmark numbers.

Test Design: 9 Bugs Dug Out of a Real Codebase

The tester selected 9 real regression issues from the private commit history of their own C++/CUDA inference engine (capy-flash), covering performance cliffs, numerical drift, and a bizarre bug that only appeared in a dual-GPU WSL2 environment. Each task was restored to the pre-fix code state, and the bug report was written the way the tester would normally file an issue: describing symptoms, data, and logs—but never revealing where the problem was located.

All tasks ran on a sandboxed Mac with no network and no CUDA toolchain. Models couldn't compile or run code—they had to reason about the cause purely by reading the code and produce a patch. Each model ran each task 3 times, with a maximum of 80 steps. Claude models ran in Claude Code, local models ran in OpenCode—the tester had tried bridging local models into Claude Code but performance was noticeably worse, so each side used the tool it was best at.

NVFP4 and 3-bit Quantization

NVFP4 is a 4-bit floating-point quantization format proposed by NVIDIA, designed specifically for inference acceleration on next-generation GPUs (such as the Blackwell architecture). It uses an E2M1 mantissa-exponent allocation (1 sign bit, 2 exponent bits, 1 mantissa bit) combined with block-level scaling factors to preserve dynamic range, and typically maintains precision better than traditional INT4 in weight quantization scenarios. In contrast, what the community usually calls "3-bit quantization" refers mostly to low-bit-width weight formats produced by post-training quantization methods like GPTQ and AWQ—higher compression, but greater perturbation of model behavior. The significance of comparing the two in this article lies in: Flash-Next uses a MoE architecture, where expert layers have large parameter counts but few are activated at a time, so scores barely change after quantizing to 3-bit; whereas the dense 27B model at the same bit width didn't lose points but needed to generate significantly more tokens to complete the same reasoning—showing that the impact of low-bit-width quantization on reasoning path length is more insidious than its impact on final answer correctness.

Scoring Mechanism: Dual Scoring and Consistency Verification

Each patch was scored twice: once by automatic scoring against a hidden answer key built from the real fix, checking whether files and functions were correct, patch precision, and several code inspection items; and once by Claude Opus 5.5 acting as a blind judge, seeing the bug report, code, the real fix labeled "one valid solution, not the only solution," and compiler output, giving a score on a 5-level scale of 1 / 0.75 / 0.5 / 0.25 / 0. Patches that failed to compile were capped at 0.25 in both scoring systems.

The per-patch correlation between the two scoring sets was r = 0.79, and model rankings were consistent. The judge was slightly stricter than the answer key and showed no favoritism toward Claude—every Claude model's "judge score minus automatic score" was negative, while Qwen models were near zero. The tester candidly noted that converting real bugs into fair evaluation tasks is very time-consuming: a single model running 9 tasks × 3 runs takes 4 to 8 hours, so the sample size is limited, with a confidence interval of about ±0.2. Two models differing by less than 0.1 points are essentially a coin flip.

Blind Judge and Answer Key

The "answer key" refers to the standard answer the evaluator compiled in advance based on the real fix, containing machine-verifiable fields such as the correct file, function, and patch structure, against which automatic scoring compares item by item. The blind judge is another large model (in this case, from the Claude family) acting as referee, giving a graded score without knowing which model produced the patch, combining the bug report, code context, the real fix labeled "one valid solution, not the only solution," and compiler output. The reason for running two scoring systems in parallel is that automatic scoring is overly harsh on patches that are written differently but equally correct, while a model judge might show preference for models of its own family. The author used the per-patch correlation coefficient r = 0.79 to quantify their consistency, and additionally checked the sign of "judge score minus automatic score" across models to verify the judge wasn't systematically favoring Claude—the results showed the difference was negative for Claude models, meaning the judge was actually stricter, providing corroborating evidence for the credibility of the scoring.

Result Tiers: Opus Leads, Flash-Next Closely Trails Haiku

The results showed Claude Opus 5.5 at the top with 0.74 points, consuming an average of 16k tokens per task and taking 3 minutes. Claude Sonnet 5.5 came second with 0.58 points, achieving this with only 8k tokens—remarkably efficient. The biggest surprise came from the local model Qwen3.8-Flash-Next: the NVFP4 quantized version scored 0.50, and the 3-bit quantized version 0.48, already surpassing Claude Haiku 5.5's 0.47 and clearly ahead of older Sonnet versions.

Claude Sonnet 5 and Sonnet 4.5 scored only 0.28 and 0.19 respectively, while Haiku 4.5 dropped to 0.13. Gemma 4 26B-A4B (thinking mode) scored just 0.07, with virtually no agent capability—with thinking off it never edited a file across all 9 tasks, and with thinking on it would edit but mostly incorrectly. Qwen3.8-27B's 3-bit and NVFP4 versions scored 0.39 and 0.34 respectively, trailing the MoE-architecture Flash-Next (paired difference +0.16, P = 0.92).

The Truth About Quantization Loss: Not Lower Scores, but Longer Thinking

The most valuable finding of this test is how quantization affects model behavior. After quantizing Flash-Next's expert layers to 3-bit, scores dropped only 0.02 and token consumption increased just 3%—essentially free at the scale of 9 tasks. But the 27B model was completely different: the 3-bit version actually scored 0.05 higher (within noise range), yet generated 40% more tokens per task (confidence interval 1.25–1.59×), and still showed 1.30× after rerunning on a new build.

The 3-bit 27B decodes faster on the tester's engine (153 vs 132 tok/s), but still needs 18 minutes instead of 13 per task. In other words, the quantization damage didn't show up in scores—it showed up in the model needing to "think longer" to reach the same place. Looking only at tok/s, the 3-bit 27B seems like the better deal; counting bugs fixed, it isn't.

This has direct implications for local deployment decisions: if your workflow is latency-sensitive, the hidden token inflation from quantization may offset the decoding speed advantage. The tester mentioned that part of the gap came from an already-fixed slow prefill path in 3-bit, but even on the new build, token consumption was still 30% higher.

Quantization Differences Between MoE and Dense Architectures

MoE (Mixture of Experts) models contain multiple expert subnetworks in each layer, but each token is routed to only a few of them, so the total parameter count can be large while the computation per activation remains relatively manageable. This sparse activation characteristic makes MoE more robust under low-bit-width quantization: quantized experts only handle part of the computation in a single inference pass, so errors don't accumulate across all parameters as they do in dense models. In this article, Flash-Next after 3-bit quantization dropped only 0.02 in score and increased token consumption by just 3%, while the dense Qwen3.8-27B under 3-bit increased token consumption by about 40%—a direct reflection of this architectural difference. For local deployers, this means "quantization loss" can't be judged only by perplexity or benchmark scores—you also need to see whether the model is forced into longer reasoning chains, which translate directly into latency and electricity costs, especially in agent-style multi-step tasks.

Efficiency Dimension: Token Consumption and Time Matter Equally

The tester particularly emphasized that tokens per task matter as much as scores. Sonnet 5.5 reached 0.58 with 8k tokens, while Flash-Next needed 53k tokens to get 0.50—converted to actual time, the former fixes a bug in an average of 1.5 minutes, the latter needs 10 minutes. For daily use, this efficiency gap may be more noticeable than a few tenths of a point in score difference.

The number of patches that failed to compile also provides some side information: Flash-Next was 5/27 under 3-bit and 1/27 under NVFP4; for the 27B it was 2/27 versus 4/27, showing no consistent pattern.

Test Boundaries and Open Questions

The tester explicitly listed the limitations: all 9 tasks came from a single codebase in a single domain (GPU inference), bug reports were written in Italian; Claude and local models used different harnesses; the judge itself is a Claude model. Since the sandbox couldn't compile or run, this test measures the ability to "read code, diagnose, and patch," not the full edit-build-test loop. The tasks themselves won't be made public, otherwise they'd lose their evaluation value.

The question he finally posed to the community is worth pondering for every local model user: if you've validated quantized models on your own real workloads rather than just looking at perplexity or KL divergence, have you observed the same pattern—similar scores, but the model thinks longer?

The sample size of this test is indeed insufficient to support general rankings, but it provides a more practical evaluation framework than tok/s: test models with real bugs while recording scores, token consumption, and actual time. For developers considering local deployment, this means quantized models can absolutely replace some cloud models on low-to-medium difficulty tasks, but the "thinking cost" needs to be factored into the ledger.

Agent Harness and Sandbox Limitations

"Harness" refers to the agent runtime framework wrapping a large model, responsible for tool calls, file editing, multi-step loops, and context management. In this article, Claude models ran in Claude Code and local models ran in OpenCode—the two differ in prompt templates, tool interfaces, and step control, which is one of the confounding factors the author explicitly listed: model performance comes partly from the model itself and partly from the harness design. On the other hand, the test sandbox deliberately stripped out the compiler and CUDA toolchain, so models could only read code statically and couldn't run verification—thus measuring the upstream ability to "read code, diagnose, and patch" rather than the full edit-build-test loop. In real development, an agent that can compile and run tests can self-correct through error feedback, while a static sandbox amplifies the difficulty of one-shot reasoning—a boundary worth noting when interpreting the scores.

One-line Summary

Developer compares local quantized models with Claude on 9 real bugs, finding the cost of quantization is longer thinking rather than lower scores.

Paragraph Summary

A developer used 9 real regression bugs from their own C++/CUDA inference engine to compare the bug-fixing capabilities of local quantized models and the Claude family in a compile-free sandbox. Claude Opus 5.5 led with 0.74, Sonnet 5.5 scored 0.58 using only 8k tokens; local Qwen3.8-Flash-Next's NVFP4 version scored 0.50, surpassing Claude Haiku 5.5's 0.47, but consumed 53k tokens. The most valuable finding: the 3-bit quantized 27B model didn't lose points but consumed about 40% more tokens per task—quantization damage manifests as the model "thinking longer" rather than worse answers. The author emphasizes that token consumption and time matter as much as scores, and notes the limited sample size doesn't constitute a general ranking.