ARC-Challenge
A scientific reasoning benchmark released by the Allen Institute for AI, consisting of approximately 1,172 multiple-choice questions selected because neither traditional retrieval methods nor word co-occurrence algorithms can solve them. The benchmark emphasizes genuine reasoning ability over surface-level pattern matching.
Timeline (last 90 days)
In the author's published 25-question small-sample evaluation, ZY-S1 improved over the base model on ARC-Challenge from 76 to 80 (+4 percentage points) and on MMLU from 48 to 52 (+4 percentage points).
A Reddit technical post claims that a model with only 0.8B parameters achieved a score of 42.15% on the ARC-Challenge benchmark, surpassing the untuned stock Qwen3.5-2B base model and outperforming a solution using SFT distillation with 25,000 data entries.
The Challenge subset of ARC specifically selects questions that both traditional IR systems and word co-occurrence models answered incorrectly, comprising approximately 1,172 test questions.
ARC-Challenge consists of four-choice multiple-choice questions, with a random baseline of 25%.
GPT-4-level models can achieve over 90% on the ARC-Challenge test, and many mainstream models above 7B can already exceed 70%.
The claim lacks support from a complete paper and open-source code, and key issues such as evaluation setup (0-shot or few-shot), fairness of comparisons, and reproducibility cannot be confirmed from the original content.
All Facts (6)
In the author's published 25-question small-sample evaluation, ZY-S1 improved over the base model on ARC-Challenge from 76 to 80 (+4 percentage points) and on MMLU from 48 to 52 (+4 percentage points).
50%UnverifiedA Reddit technical post claims that a model with only 0.8B parameters achieved a score of 42.15% on the ARC-Challenge benchmark, surpassing the untuned stock Qwen3.5-2B base model and outperforming a solution using SFT distillation with 25,000 data entries.
50%UnverifiedThe Challenge subset of ARC specifically selects questions that both traditional IR systems and word co-occurrence models answered incorrectly, comprising approximately 1,172 test questions.
50%UnverifiedARC-Challenge consists of four-choice multiple-choice questions, with a random baseline of 25%.
50%UnverifiedGPT-4-level models can achieve over 90% on the ARC-Challenge test, and many mainstream models above 7B can already exceed 70%.
50%UnverifiedThe claim lacks support from a complete paper and open-source code, and key issues such as evaluation setup (0-shot or few-shot), fairness of comparisons, and reproducibility cannot be confirmed from the original content.
50%