Neo-Classic
A benchmark designed to evaluate large language models' understanding of classical Chinese poetry. It uses contemporary expert-composed regulated verse as out-of-sample data and employs reverse-comprehension probes to test models' hierarchical constraint satisfaction and global planning capabilities.
Timeline (last 90 days)
A new study proposes a benchmark called Neo-Classic, designed to test the genuine reasoning ability of large models on classical poetry.
Neo-Classic consists of two parts: an out-of-sample (OOS) dataset and a set of reverse understanding probes.
Neo-Classic includes contemporary expert-authored poems that strictly conform to metrical rules, in order to reduce the likelihood of models retrieving answers from training data.
The research team designed five behavioral probes to test models' ability to satisfy hierarchical constraints.
When models transitioned from historical texts to contemporary texts, a performance drop of 20% to 50% was observed.
Models' standard accuracy on the passage-level ranking task was only 0% to 13%.
After introducing expert-level guided prompts, reasoning-enhanced models improved their performance on the passage-level ranking task to 36%, still with a notable gap compared to human experts.
The research concludes that models can capture local formal patterns but struggle to perform the global hierarchical planning required for robust linguistic-aesthetic reasoning.
All Facts (8)
The research concludes that models can capture local formal patterns but struggle to perform the global hierarchical planning required for robust linguistic-aesthetic reasoning.
70%UnverifiedNeo-Classic consists of two parts: an out-of-sample (OOS) dataset and a set of reverse understanding probes.
50%UnverifiedNeo-Classic includes contemporary expert-authored poems that strictly conform to metrical rules, in order to reduce the likelihood of models retrieving answers from training data.
50%UnverifiedThe research team designed five behavioral probes to test models' ability to satisfy hierarchical constraints.
50%UnverifiedModels' standard accuracy on the passage-level ranking task was only 0% to 13%.
50%UnverifiedAfter introducing expert-level guided prompts, reasoning-enhanced models improved their performance on the passage-level ranking task to 36%, still with a notable gap compared to human experts.
50%UnverifiedWhen models transitioned from historical texts to contemporary texts, a performance drop of 20% to 50% was observed.
50%UnverifiedA new study proposes a benchmark called Neo-Classic, designed to test the genuine reasoning ability of large models on classical poetry.
50%