ARC-AGI-2
An upgraded version of the ARC benchmark proposed by François Chollet for measuring the abstract reasoning and generalization capabilities of AI systems. It emphasizes compositional reasoning and reduces patterns that overlap with training data. Human solvers achieve an average accuracy above 85%, while mainstream large language models have consistently scored below 20%.
Timeline (last 90 days)
ARC-AGI-2, as an upgraded benchmark, raises the difficulty threshold and deliberately circumvents the shortcut of large language models memorizing answers through large-scale pretraining data.
ARC-AGI-2 places greater emphasis on compositional reasoning, requiring the simultaneous application of multiple rules to arrive at an answer, and deliberately reduces patterns that may overlap with training data.
On the ARC-AGI-2 test set, the system solved 230 out of 240 test tasks, with an overall accuracy exceeding 95%.
On the ARC-AGI-2 benchmark, Meta^n is the only method to score above zero.
ARC-AGI-2 is a benchmark specifically designed to resist skill memorization, ensuring that models cannot exploit previously seen patterns by rote learning.
The Chollet team continues to release new versions of ARC-AGI (such as ARC-AGI-2 and ARC-AGI-3) to increase difficulty and diversity.
All Facts (7)
ARC-AGI-2由AI安全研究员François Chollet设计，早期GPT-4在该测试上得分不足10%，人类平均得分约85%
80%UnverifiedARC-AGI-2, as an upgraded benchmark, raises the difficulty threshold and deliberately circumvents the shortcut of large language models memorizing answers through large-scale pretraining data.
50%UnverifiedARC-AGI-2 places greater emphasis on compositional reasoning, requiring the simultaneous application of multiple rules to arrive at an answer, and deliberately reduces patterns that may overlap with training data.
50%UnverifiedOn the ARC-AGI-2 test set, the system solved 230 out of 240 test tasks, with an overall accuracy exceeding 95%.
50%UnverifiedARC-AGI-2 is a benchmark specifically designed to resist skill memorization, ensuring that models cannot exploit previously seen patterns by rote learning.
50%UnverifiedOn the ARC-AGI-2 benchmark, Meta^n is the only method to score above zero.
50%UnverifiedThe Chollet team continues to release new versions of ARC-AGI (such as ARC-AGI-2 and ARC-AGI-3) to increase difficulty and diversity.
50%