Unverified50% confidenceFactExact time
Even in the Magicoder-Evol-Instruct-110K dataset labeled as "decontaminated," 52 of the 164 HumanEval problems still had docstring matches, and 8 had answer matches.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/13/2026
First Seen
Valid until: 1/11/2027
Sources
Related Entities
Related Claims
UnverifiedThe evaluation covered nine text-based scientific benchmarks, sampling 4,531 questions and 100 matched profiles, with 4,488 items completing all five conditions in full.63% similarUnverifiedSome practitioners adopt a methodology from HumansWith.AI, scanning the same 50 prompts weekly to track which subreddits are winning in AI citations.62% similarUnverifiedContamination detection is performed on each instruction set using two checks: the problem text is normalized by taking the first 80 characters of the HumanEval docstring and compared; the reference answer, after removing comments and docstrings, is checked for whether more than half of the 13-token n-grams match.62% similarVerifiedThe HumanEval benchmark contains 164 programming problems and is the mainstream standard for measuring code generation capability.60% similarUnverifiedThe question bank consists of 112 questions, each with a verifiable answer and a plausible wrong answer, covering arithmetic, unit conversion, world knowledge, dates, science, letter counting, multi-step math, brain teasers, and word games.59% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1013054API
curl https://kongchang.com/api/v1/knowledge/claims/1013054MCP
get_claim(id=1013054)