Unverified50% confidenceFactExact time
The Challenge subset of ARC specifically selects questions that both traditional IR systems and word co-occurrence models answered incorrectly, comprising approximately 1,172 test questions.
1
Sources
50%
Confidence
Long-term
Relevance
10/6/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedARC-AGI contains 400 training problems and 400 test problems, with each problem consisting of 2–3 input-output examples and one test case to be solved.74% similarUnverifiedEach ARC problem consists of pairs of input-output grid images; the test taker must infer the rule from 3–5 examples and apply it to a new input, and no two problems share the same solution rule.70% similarUnverifiedAutomated Program Repair (APR) research shows that even patch generation guided by test suites often suffers from overfitting to test cases, meaning the patch passes existing tests but introduces new errors on uncovered code paths.68% similarUnverifiedThe researchers constructed a benchmark test set called ALIGNMENTFORECASTBENCH, covering more than 5,000 prediction questions, 17 target models, 32 datasets, and 16 failure modes.67% similarUnverifiedThe test ultimately produced a sample of 174 tasks and 1,744 attempts, using an A/B comparison method with the same routing, same platform, and same time period.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/980503API
curl https://kongchang.com/api/v1/knowledge/claims/980503MCP
get_claim(id=980503)