ARC-AGI
ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence) is an AI reasoning benchmark proposed by François Chollet to evaluate the abstract reasoning and generalization capabilities of AI systems. The benchmark presents visual pattern transformation tasks requiring models to induce rules from a small number of examples and infer answers, emphasizing human-like core knowledge reasoning rather than large-scale memorization or statistical learning. It is a reference standard for measuring progress toward artificial general intelligence.
Core Facts
ARC-AGI is a benchmark designed by AI safety researcher François Chollet specifically to measure fluid intelligence.
François Chollet emphasizes abstract reasoning and generalization ability, and proposed the ARC benchmark as a measure of these capabilities.
ARC-AGI was proposed by AI safety researcher François Chollet in 2019.
Timeline (last 90 days)
Compared with benchmarks such as ARC-AGI, this framework allows finer-grained control over the dimension of novelty—whether the rule itself is new, the rule combination is new, or the rule instance is new.
ARC-AGI is a general intelligence benchmark proposed by François Chollet, comprising a series of colored grid transformation tasks that require a model to infer abstract transformation rules from very few examples and apply them to new inputs.
Chollet has repeatedly emphasized that ARC is merely a tool for measuring a certain type of abstract reasoning ability rather than the ultimate criterion for determining AGI, and that achieving a high score on ARC-AGI does not equate to achieving artificial general intelligence.
The ARC-AGI benchmark is designed to distinguish between two capabilities: 'performing well on tasks that have been seen before' and 'being able to flexibly handle entirely novel structural problems.'
Claude Opus's best score on the ARC-AGI private holdout set with native validation is 30%.
Prime Agent achieved approximately 78% on ARC-AGI using the GPT series and 95.5% using Opus.
With an appropriate Harness, Prime Agent achieved 95% on ARC-AGI.
On the ARC-AGI benchmark, the same model can jump in score from 30% to 95% or even 100% solely through different Harness designs.
The ARC-AGI benchmark is deliberately designed to be easy for humans and difficult for machines, in order to highlight the uniqueness of human cognition.
Once a benchmark is made public and repeatedly attacked, targeted overfitting may occur—that is, learning the distribution patterns of the problems without necessarily corresponding to underlying generalized intelligence.
30 more timeline events
All Facts (20)
ARC-AGI is a benchmark designed by AI safety researcher François Chollet specifically to measure fluid intelligence.
85%VerifiedFrançois Chollet emphasizes abstract reasoning and generalization ability, and proposed the ARC benchmark as a measure of these capabilities.
80%VerifiedARC-AGI was proposed by AI safety researcher François Chollet in 2019.
80%VerifiedARC-AGI specifically measures fluid intelligence, whereas traditional large language models primarily exhibit crystallized intelligence.
75%VerifiedUnder the standard framework of ARC Prize, GPT-6 Astra's ARC-AGI-3 score is only 62.7.
65%VerifiedThe ARC-AGI benchmark was proposed by François Chollet in 2019 to evaluate AI's abstract reasoning and generalization capabilities when faced with entirely novel problems.
65%VerifiedARC-AGI由AI研究者François Chollet设计，用于创造对人类直觉简单但对依赖数据记忆的语言模型极为困难的测试集
65%UnverifiedARC-AGI的每道题由彩色网格构成，要求模型从2-3个输入输出示例中归纳出变换规则再应用到新输入上
95%UnverifiedARC-AGI-2由AI安全研究员François Chollet设计，早期GPT-4在该测试上得分不足10%，人类平均得分约85%
80%UnverifiedOnce a public benchmark attracts widespread attention, it risks being targeted for optimization, and high scores do not necessarily reflect generalized reasoning ability.
60%UnverifiedARC-AGI is a general intelligence benchmark proposed by François Chollet, comprising a series of colored grid transformation tasks that require a model to infer abstract transformation rules from very few examples and apply them to new inputs.
50%UnverifiedCompared with benchmarks such as ARC-AGI, this framework allows finer-grained control over the dimension of novelty—whether the rule itself is new, the rule combination is new, or the rule instance is new.
50%UnverifiedChollet has repeatedly emphasized that ARC is merely a tool for measuring a certain type of abstract reasoning ability rather than the ultimate criterion for determining AGI, and that achieving a high score on ARC-AGI does not equate to achieving artificial general intelligence.
50%UnverifiedThe ARC-AGI benchmark is designed to distinguish between two capabilities: 'performing well on tasks that have been seen before' and 'being able to flexibly handle entirely novel structural problems.'
50%UnverifiedClaude Opus's best score on the ARC-AGI private holdout set with native validation is 30%.
50%UnverifiedOn the ARC-AGI benchmark, the same model can jump in score from 30% to 95% or even 100% solely through different Harness designs.
50%UnverifiedWith an appropriate Harness, Prime Agent achieved 95% on ARC-AGI.
50%UnverifiedPrime Agent achieved approximately 78% on ARC-AGI using the GPT series and 95.5% using Opus.
50%UnverifiedOnce a benchmark is made public and repeatedly attacked, targeted overfitting may occur—that is, learning the distribution patterns of the problems without necessarily corresponding to underlying generalized intelligence.
50%UnverifiedThe ARC-AGI benchmark is deliberately designed to be easy for humans and difficult for machines, in order to highlight the uniqueness of human cognition.
50%