CARAT Revealed: Are Materials Science LLMs Reasoning or Reciting?

The Question Hidden Behind Accuracy Numbers

When a materials science large language model (LLM) answers a question about crystal structure, is it actually reasoning from the structure itself — or is it simply copying an answer that's already printed in the input? This is the central challenge posed by the latest arXiv paper, CARAT: Do Materials LLMs Reason or Recite?

The tricky part: traditional accuracy metrics can't tell the difference. A structural description text often directly "prints" the very field being evaluated — so a model can score high just by copy-pasting the answer, even if it has no understanding of the underlying physics of crystal structure. This is the so-called "reasoning vs. reciting" problem.

How CARAT Designs a "Fair Interrogation"

The methodological core of CARAT is controlled variable experimentation. It fixes the questions and ground-truth answers, then compares across eight carefully matched "views" using a structured representational space called GraphSpace, which names each distinct structural relationship.

To fully separate recitation from reasoning, the researchers deploy a combination of techniques: matched fine-tuning, answer masking, evidence injection, paired inference, and a rule allowing the model to "refuse to answer." Together, these mechanisms form an experimental framework that can demand of a model: did you actually use the evidence?

The Value of Grounded Views

On the hardest question types in the benchmark, grounded views outperform formula-only inputs by 17.3 points. This shows that when evidence is genuinely presented in a structured way, model performance can improve significantly — provided the model is actually using that evidence.

GraphSpace is the core representational framework introduced by CARAT. It organizes atomic positions, bonding relationships, symmetry, and other crystal structure information as structured graphs, rather than outputting raw CIF files or plain text descriptions. The distinction from a plain periodic graph is that the latter only retains raw geometric and topological information, whereas GraphSpace defines different named "views" of structural relationships, making the "chain of evidence" needed for model reasoning clearly locatable. The eight views correspond to different granularities of structural description — such as local coordination environment, symmetry operations, and interlayer spacing — and each can be independently injected or masked, giving researchers precise control over what information the model can "see." Answer masking replaces the target field in the input text with a placeholder to prevent the model from directly copying it; evidence injection is the reverse operation, actively inserting structural evidence related to the answer to test whether the model can genuinely use it to reach the correct conclusion. The design logic of this combined mechanism resembles single-variable control in controlled experiments.

Turning the Lens on Itself

The most valuable part of the paper is the researchers' rigorous self-critique. GraphSpace outperforms the plain periodic graph by 19.3 points — which sounds impressive — but when the authors break it down, they find this gap is actually the product of two overlapping effects:

• When plain rendering already contains all the information the question requires, GraphSpace leads by only 1.96 points;
• When plain rendering completely omits those fields, the lead jumps to 46.7 points.

In other words, most of that eye-catching 19.3-point gap measures "what the baseline was missing," not "how much better the evidence was presented." This is a remarkably honest self-correction, and a warning to the entire field not to be misled by surface-level numbers.

Attacking Their Own Benchmark

The researchers then did something counterintuitive — they actively attacked the benchmark they had designed. They found that a shortcut rule of "skip the relationship link and read the list directly" could correctly answer four out of seven reinforced question families. So they iteratively rebuilt the benchmark until eleven such shortcuts were all pushed down to near-random-guess performance.

The most alarming finding came from the paired experiments: a frozen model would cite a relationship link, but when the researchers redirected the link elsewhere, the model gave the same answer in 95.6% of paired cases. This means the model was only "reciting" the relationship — it was never actually "using" it.

Reasoning Can Be Learned

The good news: after matched supervised training, model performance rose to 99.8%; but once the link was removed, accuracy plummeted to 23.4% — even lower than the 27.0% achievable by the best shortcut. Both steps are "learnable": the model can learn to genuinely rely on evidence, and to become appropriately fragile when evidence is absent. This is precisely what proves it has started to "reason" rather than "recite."

Matched fine-tuning refers to fine-tuning with training samples that exactly correspond to the test views, ensuring that training and test sets maintain consistent information structure and eliminating the confounding effect of distribution shift. This differs from typical fine-tuning, where training data may come from different rendering styles or field combinations, causing the model to transfer a slightly misaligned distribution at test time rather than truly learning the target task's reasoning chain. Paired inference constructs two versions of the input for the same question: one containing a real relationship link, and another with the link redirected to an irrelevant location. The model's answers under both inputs are then compared. If the model is genuinely reasoning from evidence, its answer should change when the link changes; if the answer is nearly the same, the model is simply reciting a memorized pattern. The 95.6% invariance rate was derived precisely through this paired comparison — a key metric for measuring "reasoning depth" rather than just "accuracy."

Implications for AI Evaluation

CARAT's significance extends well beyond materials science. It exposes a pervasive evaluation trap: high accuracy may simply be a product of information leakage, not genuine reasoning ability. In any domain where LLM evaluation involves input text that contains the answer field, the same "recitation illusion" may be at work.

The paradigm this work establishes — fixing question-answer pairs, controlling views, injecting and masking evidence, and being willing to attack one's own benchmark — offers a reusable toolkit for building more trustworthy AI capability assessments. In an era where LLM evaluation is increasingly consequential, this attitude of "self-falsification" in research is especially valuable.

Information leakage (data leakage) is a systemic risk in LLM evaluation, and is particularly acute in structured scientific data domains. Standard representation formats for crystal structures (such as CIF files) typically include multiple attribute fields in the same text block — space group, lattice parameters, atomic coordinates, and so on. When an evaluation task asks the model to "predict" one of these fields, and the input happens to contain that field (or equivalent information from which it can be directly derived), the model effectively only needs to perform string matching rather than physical reasoning. The same issue exists in protein structure prediction, chemical property prediction, and other areas, and becomes increasingly difficult to detect as LLM pretraining corpora expand — a model may have "seen" the complete fields of a given data point during pretraining, leading to test set contamination. CARAT's approach suggests that designing trustworthy evaluations requires not only removing the answer from the input, but also actively verifying whether the model is truly relying on the remaining information, rather than retrieving the answer from parametric memory.