Unverified50% confidenceFactExact time
Traditional unit test benchmarks such as HumanEval and MBPP differ from robustness evaluations, which require constructing adversarial inputs (messy naming, mixed responsibilities, comments that do not match the implementation, etc.).
1
Sources
50%
Confidence
Long-term
Relevance
10/20/2026
First Seen
Sources
Related Entities
Related Claims
Unverified当前主流基准测试体系如HumanEval、MBPP等代码评估集往往关注平均性能，对分布式的场景特定退化现象敏感度不足76% similarUnverified传统代码评测基准（如HumanEval、MBPP）主要测试模型在孤立函数级别的生成准确率75% similarUnverifiedStandardized AI benchmarks (such as HumanEval and SWE-bench) suffer from distortion due to benchmark overfitting.74% similarUnverifiedCharacterization tests compare byte-level output, so any change unrelated to business logic (timestamps, random IDs, container iteration order) is misjudged as a regression.74% similarUnverifiedDifferential testing (contrastive testing) constructs input pairs that differ along a single dimension such as the section they belong to, and asserts that the system gives different answers, which exposes systemic defects at the retrieval level better than ordinary QA accuracy tests73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1077159API
curl https://kongchang.com/api/v1/knowledge/claims/1077159MCP
get_claim(id=1077159)