Same Quantity, Different Answer: The Numerical Representation Invariance Flaw in Language Models

A Overlooked Problem: Change the Format, Change the Answer

For humans, 0.5, 1/2, 50%, and "one half" all refer to the same value and should yield the same answer when solving a problem. But do large language models hold up to the same standard? A recent arXiv paper, Same Quantity, Different Answer, systematically examines this deceptively simple yet diagnostically powerful question — numerical representation invariance.

The core hypothesis is straightforward: math word problems with numerically equivalent quantities should produce the same correct answer, regardless of whether a value is written as a decimal, fraction, percentage, number word, scientific notation, or a precisely converted unit. If a model's answers drift across these equivalent representations, it reveals a dependence on surface form rather than genuine numerical understanding.

Experimental Design: 3,600 Problems and Five Invariance-Preserving Transformations

The research team generated 3,600 exact rational-number problems and expanded them into 8,600 prompts covering five families of "identity-preserving" transformations — that is, transformations that change only how a value is written, not its actual numerical value. Five open-weight systems were evaluated.

To avoid introducing noise at the evaluation stage, the team employed a fixed syntactic audit pipeline that normalizes common answer formats without relying on an LLM judge. This design is particularly clever because it separates two things that are often conflated: model reasoning ability and evaluator parsing ability.

Open-weight models are those whose weight parameters are publicly released and can be deployed locally, as opposed to fully closed commercial API models. Using open-weight models in evaluation research allows researchers to fix a model snapshot and ensure exact reproducibility, without worrying about silent server-side updates causing result drift.

The five identity-preserving transformation families are: decimal notation (0.5), fractions (1/2), percentages (50%), English number words (one half), scientific notation (5×10⁻¹), and equivalent unit conversions (e.g., converting kilometers to meters). These transformations share a common trait — they alter the visual presentation of a value without touching its mathematical essence. Any system that truly understands quantities should produce stable answers across all of them.

Key Findings: Invariance Collapses Behind High Accuracy

On canonical accuracy, the five systems performed impressively, ranging from 0.969 to 0.996. But once representational transformations were introduced, things deteriorated sharply:

• Orbit correctness dropped to 0.848–0.981
• Orbit invariance fell to 0.851–0.981
• "Invariant but incorrect" orbits were extremely rare, at most 0.003

In other words, models produced inconsistent answers when the same number was written in different ways. The source of these failures deserves closer scrutiny.

Most "Failures" Are the Evaluator's Fault

The study found that widespread failures under the strict parser were mostly not due to model reasoning errors — they stemmed from multiplicative-form scientific notation (e.g., 3 × 10^4) falling outside the implemented numerical grammar. In other words, the model's answer was correct; the evaluator simply couldn't recognize the format and misclassified it as wrong.

This is one of the paper's most important warnings: evaluator interface limitations masquerade as reasoning failures. When interpreting benchmark results, conflating the evaluation pipeline with model capability can easily lead to misleading conclusions.

"Orbit" is the central conceptual unit of this study — it refers to the group of prompt variants produced by applying all equivalent transformations to a single original problem. For example, a problem containing "0.5 kilograms" has an orbit that includes versions rewritten as "1/2 kilograms," "50% kilograms," and so on.

Orbit correctness measures whether all variants within an orbit yield correct answers. Orbit invariance is more lenient — it only requires that answers within an orbit be mutually consistent (even if the entire orbit is answered incorrectly, it still counts as "invariant"). The distinction between these two metrics reveals different failure types: the former conflates reasoning errors with inconsistency errors, while the latter isolates sensitivity to surface form.

The near-zero rate of "invariant but incorrect" orbits (≤0.003) indicates that when models err, they tend to err inconsistently — different surface forms trigger different mistakes, rather than a stable, uniform error.

A Genuine Semantic Pathology: Mistral Small 4's Unit Conversion Errors

After filtering out evaluator artifacts, one genuine semantic issue remained. Mistral Small 4 scored only 0.699 on unit-conversion inputs, producing 265 errors where the wrong answers differed from the correct labels by exactly a power of ten.

This kind of systematic "off-by-an-order-of-magnitude" error is highly diagnostic. It doesn't look like random noise — it points to a stable cognitive bias in how the model handles decimal placement or magnitude during unit conversion. Such errors are particularly dangerous in real-world applications because the answers look structurally plausible, yet are off by one or more orders of magnitude.

Representation Consensus vs. Paraphrase Consensus: Majority Voting Isn't a Silver Bullet

The paper also designed an independent 9,000-call experiment, allocating equal call budgets across comparison groups, to test whether representation consensus (voting across multiple numerical representations of the same problem) outperforms traditional paraphrase consensus (rephrasing the same problem in different natural language forms).

The conclusion was surprising: on low-error-rate subsets, representation consensus did not outperform paraphrase consensus — it actually produced significantly more false alarms. This warns practitioners that simply increasing numerical representation diversity for ensemble voting does not reliably improve robustness, and can sometimes backfire.

Paraphrase consensus is a common variant of self-consistency sampling: the same problem is posed multiple times using different natural language phrasings, and a majority vote over the responses reduces random variance. Popularized by Wang et al. (2022) in chain-of-thought reasoning research, this approach has effectively improved accuracy on multiple benchmarks.

Representation consensus is this study's analogous proposal: instead of varying the linguistic phrasing, vary the numerical representation format. In theory, if a model's responses to different numerical representations involve independent random errors, majority voting should suppress noise. But experimental results show that models exhibit systematic biases toward certain representation formats rather than random errors — meaning voting amplifies false alarms, because problematic formats consistently "steer" the model toward wrong answers.

Reproducibility: A Complete Frozen Benchmark and Toolchain

The research includes a companion archive containing the frozen benchmark set, evaluation and audit logs, raw responses from the consensus experiments, manifest files, analysis code, and a single-command paper build script. This level of reproducibility is itself a demonstration of best practices for community evaluation standards.

Implications for Practitioners

The value of this work lies not in exposing some catastrophic flaw, but in providing a methodology for distinguishing real problems from phantom problems:

• Don't mistake evaluator parsing limitations for model reasoning failures, especially when scientific notation, units, or non-standard formats are involved.
• Watch out for systematic order-of-magnitude errors — they are more insidious and more dangerous than random errors.
• Design ensemble voting carefully — increasing input diversity doesn't necessarily improve consistency and can sometimes make things worse.

For any application that relies on language models to handle numerical and computational tasks, representation invariance should be a standard check on the evaluation checklist.