Neo-Classic Benchmark: Can LLMs Truly Understand Classical Chinese Poetry?

High Scores Don't Mean True Understanding

Large language models (LLMs) tend to achieve impressively high accuracy on existing classical Chinese poetry benchmarks — but a fundamental question remains unanswered: have these models actually acquired transferable "linguistic-aesthetic reasoning" abilities, or are they simply reciting answers memorized from familiar training data? This question exposes a core limitation of current evaluation methods: when tests are built from historical texts, it's nearly impossible to tell whether a model is reasoning or retrieving from memory.

To address this problem, a new study introduces a benchmark called Neo-Classic. Its central idea is to shift test content away from historical texts the model may have "memorized" toward contemporary works it almost certainly hasn't seen before — enabling a purer assessment of genuine reasoning ability.

The Design Ingenuity of Neo-Classic

Neo-Classic consists of two components: a constructivist Out-of-Sample (OOS) dataset, and a set of reverse understanding probes.

Unlike traditional benchmarks that rely on validating or generating from historical corpora, Neo-Classic exclusively uses contemporary expert-composed poetry that strictly adheres to classical prosodic rules. The key advantage here is dramatically reducing the possibility that a model can "retrieve" answers directly from its training data. In other words, models face entirely new text that still follows rigorous classical rules — and can only answer correctly by genuinely understanding those rules.

The research team designed five behavioral probes specifically to test a model's ability to satisfy hierarchical constraints. The rules of classical poetry are far from flat — from individual characters and tonal patterns to parallelism and overall structure, multiple layers of constraints interact — making it an ideal setting to distinguish between "local pattern recognition" and "global planning ability."

---

Hierarchical constraint satisfaction is the defining feature of classical Chinese prosody and the theoretical foundation of this evaluation framework. Take regulated verse (律诗) as an example: constraints are nested from the micro to the macro level. At the lowest level are individual character tones (level/oblique); above that are the tonal patterns within each line (e.g., 仄仄平平仄仄平); above that are the 粘对 rules governing tonal cohesion between adjacent couplets (粘: the opening line of one couplet matches the tonal pattern of the closing line of the previous; 对: the two lines within a couplet are tonally opposite); and at the top level are the overall structural arc, unified imagery, and emotional progression of the entire poem. This mutually dependent multi-layer constraint structure means that generating or evaluating a poem requires not just checking local tonal correctness, but coordinating all constraints at a global level — which is precisely what makes classical poetry an ideal testbed for evaluating LLMs' global planning capabilities.

---

How Three Top Models Actually Performed

The study evaluated several state-of-the-art models, including Qwen3-Max, Gemini-3-Pro, and DeepSeek-V3.2. The results revealed two major limitations.

Limitation 1: The Performance Gap from Historical to Contemporary Texts

When models were switched from historical to contemporary texts, performance dropped by 20% to 50%. This significant gap is compelling evidence that the high scores models achieve on historical corpora rely substantially on familiar pattern matching rather than genuinely transferable reasoning. Once presented with unseen contemporary regulated poetry, the models' limitations are laid bare.

---

This phenomenon is commonly known in machine learning as data contamination or benchmark leakage: the pretraining corpus already contains the test questions or content highly similar to them, causing inflated evaluation scores that don't reflect true generalization ability. For classical poetry tasks specifically, the internet contains vast amounts of annotations, literary analyses, and quiz banks covering canonical historical works — meaning models very likely encountered relevant answers during pretraining. This is precisely the value of the Out-of-Sample design: by using contemporary works created after a model's training cutoff or from controlled sources, researchers can largely eliminate the confounding effect of memory retrieval, making performance numbers a closer approximation of true linguistic-aesthetic reasoning ability.

---

Limitation 2: Complete Collapse on Passage-Level Ordering Tasks

The more severe problem emerged in passage-level ordering tasks. Models' standard accuracy on these tasks was extremely low — just 0% to 13%. This means that when required to understand and organize the global structure of a poem, grasping the logical and aesthetic ordering between lines, current models are nearly helpless.

Even with expert-level guided prompting, reasoning-enhanced models only improved to 36% — still a significant gap from human expert performance.

Local Patterns vs. Global Planning

These findings collectively outline the capability boundary of current LLMs in classical Chinese poetry understanding: models can capture local formal patterns, but struggle to perform the global hierarchical planning required for robust linguistic-aesthetic reasoning.

To put it another way, these models resemble a test-taker who has memorized tonal rules and parallelism formulas — capable of handling isolated rule checks, but unable to truly "compose a poem from the top down." The holistic grasp of overall imagery, structural echoes, and emotional progression that characterizes both the creation and appreciation of classical poetry is precisely where hierarchical reasoning ability is most concentrated — and where current models are weakest.

Neo-Classic's value lies not just in producing a set of scores, but in offering a more honest evaluation paradigm. By combining out-of-sample data with reverse probes, it separates "memory" from "reasoning" as much as possible. For researchers who care about the true cognitive capabilities of LLMs, this type of benchmark helps avoid being misled by inflated scores on historical corpora.

Implications for AI Capability Evaluation

The significance of this research extends well beyond classical Chinese poetry. It reminds us that in any domain with rich historical corpora, a model's high scores may be partly attributable to data leakage or pattern memorization. To truly measure a model's transfer and reasoning abilities, we need to construct out-of-sample, retrieval-resistant evaluation tasks.

Classical Chinese poetry, with its strict hierarchical prosody, serves as a natural touchstone for testing global planning ability. And the 0%–13% passage ordering accuracy makes one thing clear: between "locally correct" and "globally coherent," today's language models still face a chasm they have yet to cross.