How SWE-bench Detects Cheating: Similarity Analysis Between Submissions and Ground Truth Patches

Background: The Credibility Problem in AI Coding Benchmarks

SWE-bench is one of the most authoritative benchmarks for evaluating large language models' software engineering capabilities. Many AI coding products and models use their performance on this leaderboard as a core selling point. However, a critical question has always lingered: Are the model solutions submitted to the leaderboard truly "independently solved," or do they somehow "copy" from the ground truth patches?

SWE-bench team member John Yang recently published a detection script and its analysis results, systematically measuring the similarity between all submissions and the ground truth (gold patches), and uncovering one anomalous case. This work provides an important safeguard for benchmark transparency and credibility.

Detection Method: The Technical Principles of Per-Hunk Exact Matching

Core Approach

Simply comparing model-generated patches with gold patches as strings doesn't work, because patch files contain extensive metadata (such as line numbers, context information, etc.). John Yang designed a more refined detection pipeline:

1. Remove comments: Strip all comment lines from both gold patches and predicted patches, focusing only on actual code changes
2. Filter irrelevant files: Ignore files in the predicted patch that weren't edited in the gold patch
3. Hunk-level matching: Parse patches into unidiff.PatchSet objects, then check whether each hunk (code change block) in the gold patch appears verbatim in the predicted patch

Judgment Criteria

If all hunks from the gold patch can be found with exact matches in the predicted patch, that prediction is flagged as an "exact match." Here, "exact match" means the gold answer is verbatim contained within the model's output — a fairly strict standard, meaning the model not only solved the problem but did so in a way completely identical to the human developer's PR.

The elegance of this approach lies in the fact that it doesn't require the predicted patch to be identical to the gold patch (the model may have made additional modifications). Instead, it checks whether the gold answer is "embedded" within the model's output.

Analysis Results: The Vast Majority of Models Behave Normally

Exact Match Rate Data Across Subsets

John Yang analyzed all submissions across SWE-bench's three subsets (Verified, Lite, Test), with the following results:

Subset	Average Exact Match Rate	Range	Typical Count
SWE-bench Verified	6.7%	0% - 13%	~34/500
SWE-bench Lite	4%	0% - 11.2%	~12/300
SWE-bench Test	2.45%	0% - 4.05%	~56/2294

Note: The Verified statistics exclude one outlier.

These numbers send a positive signal: the vast majority of model solutions are genuinely "original" rather than simple regurgitations of the gold answer. The 6.7% average exact match rate means that out of 500 problems, models on average produce solutions identical to the gold answer for only about 34 problems — considering that some simple bug fixes inherently have very limited repair approaches (such as changing a variable name or adjusting a conditional), this ratio is reasonable.

Relationship Between Subset Size and Match Rate

An interesting observation: the smaller the problem set, the higher the exact match rate. Verified (500 problems) shows 6.7%, Lite (300 problems) shows 4%, while the full Test set (2294 problems) shows only 2.45%. This likely reflects a pattern: smaller subsets tend to include more "classic" or "typical" bugs whose fixes are more deterministic, making it more likely for models to "converge" on solutions identical to the gold answer.

The Honeycomb Anomaly: 87% Exact Match Rate

Discovering the Anomaly

Among all submissions, one named 20240820_honeycomb drew significant attention: its exact match rates on Verified and Lite were 78.7% and 87.2% respectively — far exceeding the normal range. Even more suspicious, its exact match rate was far higher than its actual resolution rate, which is logically very abnormal.

Investigation Conclusion: Submission Format Error, Not Deliberate Cheating

Deeper investigation revealed that the submission's prediction file contained 2,236 predictions, far exceeding the Verified subset's 500 instances. This was most likely a human error: a file prepared for the full test set was mistakenly uploaded as a Verified submission.

Key verification steps:

• Re-filtering: After keeping only the 500 instances belonging to Verified, the exact match rate dropped to 16/500 (approximately 3.2%), completely normal
• Cross-validation: Honeycomb's separate submission on the full Test set had an exact match rate of only 1.7%, also normal
• Anomaly source: The extra 1,736 instances had extremely high exact match rates; the most reasonable explanation is that these weren't actual model predictions but accidentally included gold answers or close copies thereof

Final conclusion: This was a submission format error, not deliberate cheating. However, this case served as an excellent stress test for the detection mechanism, proving that the method can effectively identify anomalies.

It's worth noting that this submission had previously been removed for other reasons, and the team had contacted the submitter requesting a technical report but received no response.

Future Plans and Industry Significance

Routine Detection Mechanism

The SWE-bench team stated that they will routinely run this detection script on all future submissions, and will require additional explanations from submitters whose exact match rate exceeds 20%. This threshold is set with reasonable margin — the highest value among normal submissions is approximately 13%, so the 20% threshold can catch genuine anomalies without generating too many false positives.

Implications for Anti-Cheating in AI Benchmarks

The significance of this work extends beyond SWE-bench itself. As competition among AI models on various benchmarks intensifies, anti-cheating mechanisms for benchmarks become increasingly important:

• Data contamination issues: Has the model seen the test set's gold answers in its training data? Exact match detection can serve as an indirect signal
• Transparency standards: Publishing detection methods and results builds a foundation of trust for the entire community
• Methodological reference: Other benchmarks (such as HumanEval, MATH, etc.) can also draw on similar detection approaches

In an era of rapidly advancing AI capabilities, ensuring the fairness and credibility of evaluation systems is just as important as improving model capabilities themselves. The SWE-bench team's initiative sets a benchmark worth promoting across the industry.