7 AI Models Fix the Same Bug: GLM 5.1 Overtakes Claude Sonnet in Detailed Comparison

Test Background: Real PR Reproductions from OpenClaw, a 350K-Star Project

This is a unique AI programming showdown. The tester selected three recently merged PRs from OpenClaw (a 350K-star open-source project) and had 7 AI models independently fix bugs without knowing the answers. The solutions were hidden in already-merged branches, serving as the final judge.

Notably, OpenClaw itself was entirely built by the author Peter using AI web coding—PRs were submitted by AI, reviews were done by AI, and merging to main was decided by AI. So the essence of this test is: 7 AIs fixing bugs written by AI, with humans doing only one thing throughout—paying for the coding plan.

OpenClaw represents the new paradigm of "AI-Native Development." In this model, AI isn't just an assistive tool but handles the complete development loop from code generation, PR submission, Code Review, to branch merging. This is fundamentally different from traditional "AI-assisted programming" (like GitHub Copilot's code completion)—the latter still has humans as the decision-making core, while in AI-native development, humans have devolved into a "resource provider" role. The viability of this model depends on LLMs' comprehensive capabilities in code understanding, contextual reasoning, and engineering judgment, which also makes "AI fixing bugs written by AI" a self-referential capability test. This test also adopted a "real PR reproduction" methodology, which has higher ecological validity compared to traditional programming contests (like LeetCode) or synthetic benchmarks (like HumanEval or SWE-bench)—using "recently merged PRs" as the test set effectively mitigates training data contamination risks and more closely approximates quality standards in real engineering scenarios.

Contestants and Scoring Rules

For the international contestant, Claude Sonnet 4.6 was deliberately chosen over Opus 4.7, reasoning that Opus is too strong for meaningful comparison—Sonnet is the baseline that domestic open-source models should be catching up to.

The 6 domestic open-source contestants:

• DeepSeek V4 Flash: Pro was eliminated last episode, Flash steps in as substitute
• Kimi K2.6: Veteran contestant since the first episode
• GLM 5.1: Previously a stable runner-up, can it go further this time?
• MiniMax M2.7: The quality inspector role, consistent participant
• Mimo V2.5 Pro: Resilient under criticism, seeking redemption
• Qwen 3.6 Plus: Consecutive danger warnings, can the luck hold?

Scoring dimensions cover five aspects: passing tests, correctness of fix, cleanliness of changes, supplemental tests, and final weighted composite score.

Easy Problem: Crown Twist Lack Notification Failure

Problem Analysis

This issue practically gave away the answer—the test was who could achieve the most minimal change. The official approach passes AllowBootstrap as a parameter, only setting the handler in the Crown path.

Result: Everyone Converged

All 7 models independently used the same simplified approach—hardcoding directly at the source. Functionally equivalent but more aggressive, affecting all callers. This "lazy" approach solves the problem but is less elegant than the official solution.

The differences emerged in test coverage, and test coverage itself is a core engineering metric for measuring code quality. When LLMs generate test code, they actually need to perform reverse reasoning on the code under test, enumerating boundary conditions and exception paths. Google's engineering practices show that the cost of fixing production bugs is 6-10x that of the development phase, and adequate unit testing can reduce regression rates by over 40%. In this problem:

• GLM 5.1 added 19 lines of tests
• Kimi K2.6 added 12 lines
• The other five (including Sonnet 4.6) wrote zero test lines

Scores for this problem: GLM 5.1 (85) > Kimi K2.6 (80) > Everyone else (75)

Medium Problem: MCP Protocol Client Signal Forwarding

Problem Analysis

A Signal needs to be forwarded between two layers—this is an MCP protocol client's about-after-plugin-drag-won't-stop issue.

MCP (Model Context Protocol) is an open standard protocol proposed by Anthropic in late 2024, aimed at solving the integration fragmentation between AI models and external tools/data sources. In MCP architecture, there's a three-layer structure of Host (host application), Client, and Server (tool service endpoint). The Signal forwarding problem is a typical asynchronous event propagation scenario—when a user initiates a cancellation request, the signal needs to penetrate from the top-level Host through the Client layer, ultimately reaching the Server-side tool call being executed. If the Signal is lost at the intermediate layer, long-running tools cannot respond to cancel commands, causing resource leaks and UX issues. This problem tests precisely the model's depth of understanding of event propagation mechanisms in layered protocol architectures.

Result: Everyone Correct, Gaps in Test Coverage

All 7 models fixed it using the exact same pattern as the official solution:

1. Add signal parameter to call
2. Pass signal through in toexecute
3. Get signal from extra in set request's handler

Three steps, not a single character different. This strongly suggests the original PR was also submitted by AI and merged after AI review.

Test coverage gaps are the real watershed:

• GLM 5.1: 108 lines of tests, 1.3x Sonnet's, 2x Kimi's
• MiniMax: 87 lines, more than Sonnet
• Kimi: Only 55 lines, the veteran slacked off

Scores for this problem: GLM 5.1 (92, near-perfect) > MiniMax/Sonnet (90) > Qwen (87) > Kimi (84)

Hard Problem: Gateway Startup Validation Too Strict

Problem Analysis

Fatal errors fire before plugins are loaded—need to distinguish between "plugin not installed" and "installed but not yet started," handling each case differently. The Issue provided three approaches; the core is distinguishing stale evidence from missing plugin.

This problem involves an important defensive programming pattern: Evidence-Based State Discrimination. In plugin system design, "plugin not installed" and "plugin installed but not started" are two fundamentally different system states corresponding to different error handling strategies. The former is a configuration absence that should block the startup flow at Fatal level; the latter is a timing issue that can be downgraded to Warning while waiting for the plugin to become ready. This reflects an error-handling variant of the "Principle of Least Privilege"—error severity should precisely match its actual impact scope. Overly lenient error handling can mask real system faults and increase troubleshooting difficulty.

Result: 6 Correct, 1 Wrong

6 models used a gating function hadStalePluginEvidence, only downgrading severity when plugin evidence actually exists—consistent with the official solution.

Qwen 3.6 Plus was the only failure: It directly changed error push to warning push, downgrading everything regardless of whether the plugin was installed—violating the Issue's explicit requirements, scoring only 50. This is a textbook example of ignoring the "Evidence-Based State Discrimination" principle—using a blanket lenient approach to mask real configuration-absence scenarios.

Sonnet 4.6 scored the highest of the entire test at 93: Most complete comments, most rigorous tests, and the most elegant implementation with gating plus early return. Three domestic open-source models scored 90 in a siege (DeepSeek, GLM, MiniMax)—only a 3-point gap, but the gap definitely exists.

Overall Rankings and Deep Analysis

Final Rankings

Rank	Model	Composite Score	Highlight
1	GLM 5.1	89.3	Test coverage dominates the field
2	Sonnet 4.6	87.2	Highest single-problem score of 93
3	DeepSeek V4 Flash	84.7	Fastest at 7 min 30 sec
Honorable	MiniMax M2.7	86.0	Best value for money
Last	Qwen 3.6 Plus	69.0	Hard problem failure, eliminated

Key Findings