Grok 4.20
A large language model in the Grok series released by xAI. The 4.20 version includes a non-reasoning variant and features multimodal understanding capabilities.
Timeline (last 90 days)
The benchmark covers 9 models from 6 labs, including Qwen3 235B, Claude Haiku 4.5, Claude Opus 5.5, Gemini series, GPT-5.4 series, GLM-5, and Grok 4.20, of which 4 are reasoning models.
The letter-counting task broke nearly all models, with Claude Haiku 4.5 holding firm only 8.3% of the time and Grok holding firm 25% of the time.
In the hard-to-verify category with no bait, GPT-5.4 nano answered only 19/40 questions correctly in both orderings and had a 65% probability of selecting Response A; GPT-5.4 mini had a 66% probability of selecting A, Grok 4.20 non-reasoning was 62%, while strong judges were stable at 50%.
Under injection bait, weak models treat the injected incorrect numbers as their own calculation results and fabricate verification processes out of thin air, with the explanatory text appearing consistent with genuine verification, making it impossible to detect errors through auditing.
After enabling reasoning mode, Grok 4.20's hard-to-verify injection score rose from 0.075 to 0.925 (nearly immune), but Qwen3-Next 80B Thinking only rose from 0.00 to 0.35, and the reasoning cost is 5 to 9 times that of non-reasoning.
Placing verbose padding on the correct answer causes the judge's score to plummet—Grok non-reasoning hard-question scores dropped from 0.875 to 0.19—indicating that verbosity bias exists only in reverse.
All Facts (6)
The benchmark covers 9 models from 6 labs, including Qwen3 235B, Claude Haiku 4.5, Claude Opus 5.5, Gemini series, GPT-5.4 series, GLM-5, and Grok 4.20, of which 4 are reasoning models.
50%UnverifiedThe letter-counting task broke nearly all models, with Claude Haiku 4.5 holding firm only 8.3% of the time and Grok holding firm 25% of the time.
50%UnverifiedIn the hard-to-verify category with no bait, GPT-5.4 nano answered only 19/40 questions correctly in both orderings and had a 65% probability of selecting Response A; GPT-5.4 mini had a 66% probability of selecting A, Grok 4.20 non-reasoning was 62%, while strong judges were stable at 50%.
50%UnverifiedUnder injection bait, weak models treat the injected incorrect numbers as their own calculation results and fabricate verification processes out of thin air, with the explanatory text appearing consistent with genuine verification, making it impossible to detect errors through auditing.
50%UnverifiedAfter enabling reasoning mode, Grok 4.20's hard-to-verify injection score rose from 0.075 to 0.925 (nearly immune), but Qwen3-Next 80B Thinking only rose from 0.00 to 0.35, and the reasoning cost is 5 to 9 times that of non-reasoning.
50%UnverifiedPlacing verbose padding on the correct answer causes the judge's score to plummet—Grok non-reasoning hard-question scores dropped from 0.875 to 0.19—indicating that verbosity bias exists only in reverse.
50%