SlopCodeBench
A benchmark for evaluating the robustness and overall performance of AI coding models when given low-quality, messy code as input, emphasizing code comprehension and refactoring capability in realistic legacy-code scenarios.
Timeline (last 90 days)
These SlopCodeBench results come from an October run of a single benchmark, with certain limitations in sample scope and task types; the relationship between reasoning effort and code quality may vary by model version and task domain.
The SlopCodeBench October test report shows that after adjusting reasoning effort from medium to xhigh, cost rose to 2 to 4 times, while the number of checkpoints passed did not improve reliably.
SlopCodeBench is a benchmark for code generation tasks that uses a checkpoint mechanism to evaluate whether model outputs satisfy a series of functional and quality requirements.
On checkpoint pass rate, xhigh showed no statistically reliable improvement; on some tasks the two were on par, and on individual tasks xhigh was even slightly lower.
The checkpoint mechanism decomposes programming tasks into a number of independently verifiable functional points or quality requirements, with each checkpoint corresponding to an explicit assertion, and evaluation is binary and cumulative.
The report points out that the extra cost brought by xhigh does not correspond to a reliable gain in pass rate, constituting diminishing marginal benefit and even being negative in some scenarios.
SlopCodeBench measures a model's overall performance under real, messy, low-quality (slop) code inputs, rather than only single-turn code generation accuracy.
The title of the SlopCodeBench 2026 leaderboard includes "xhigh considered harmful", meaning the highest reasoning effort level (xhigh) lowers the final score.
The discussion of SlopCodeBench on Hacker News accumulated only 18 points and 2 comments, with limited public details.
In SlopCodeBench, Opus 5.5 and Astra each have their wins, showing that a single model struggles to dominate all task types.
1 more timeline events
All Facts (11)
These SlopCodeBench results come from an October run of a single benchmark, with certain limitations in sample scope and task types; the relationship between reasoning effort and code quality may vary by model version and task domain.
50%UnverifiedThe SlopCodeBench October test report shows that after adjusting reasoning effort from medium to xhigh, cost rose to 2 to 4 times, while the number of checkpoints passed did not improve reliably.
50%UnverifiedSlopCodeBench is a benchmark for code generation tasks that uses a checkpoint mechanism to evaluate whether model outputs satisfy a series of functional and quality requirements.
50%UnverifiedOn checkpoint pass rate, xhigh showed no statistically reliable improvement; on some tasks the two were on par, and on individual tasks xhigh was even slightly lower.
50%UnverifiedThe checkpoint mechanism decomposes programming tasks into a number of independently verifiable functional points or quality requirements, with each checkpoint corresponding to an explicit assertion, and evaluation is binary and cumulative.
50%UnverifiedThe report points out that the extra cost brought by xhigh does not correspond to a reliable gain in pass rate, constituting diminishing marginal benefit and even being negative in some scenarios.
50%UnverifiedSlopCodeBench measures a model's overall performance under real, messy, low-quality (slop) code inputs, rather than only single-turn code generation accuracy.
50%UnverifiedThe title of the SlopCodeBench 2026 leaderboard includes "xhigh considered harmful", meaning the highest reasoning effort level (xhigh) lowers the final score.
50%UnverifiedIn SlopCodeBench, Opus 5.5 and Astra each have their wins, showing that a single model struggles to dominate all task types.
50%UnverifiedThe October 2026 SlopCodeBench leaderboard shows Opus 5.5 and Astra in leading positions, with Astra taking first place in the overall score.
50%UnverifiedThe discussion of SlopCodeBench on Hacker News accumulated only 18 points and 2 comments, with limited public details.
50%