Same Quantity, Different Answer
A research paper published on arXiv that systematically examines deficiencies in the numerical representation invariance of large language models, investigating the consistency of model answers when the same numerical value is presented in different written forms.
Timeline (last 90 days)
The arXiv paper 'Same Quantity, Different Answer' systematically examines the numerical representation invariance problem in large language models.
The research team generated 3,600 exact rational number questions and expanded them into 8,600 prompts, covering five identity-preserving transformation families.
The study evaluated five open-weight model systems.
The five systems achieved standard answer accuracy ranging from 0.969 to 0.996.
After introducing representational transformations, orbit accuracy dropped to 0.848–0.981 and orbit invariance dropped to 0.851–0.981.
Most of the widespread failures produced by the strict parser do not stem from model reasoning errors, but rather because multiplicative-form scientific notation falls outside the implemented number grammar.
Flaws in the evaluator interface can masquerade as reasoning failures, and failing to distinguish between the evaluation pipeline and model capability when interpreting benchmark results may lead to misleading conclusions.
The study is accompanied by a companion archive containing a frozen benchmark test set, evaluation audit logs, raw responses from the consensus experiments, and analysis code.
All Facts (8)
The arXiv paper 'Same Quantity, Different Answer' systematically examines the numerical representation invariance problem in large language models.
50%UnverifiedThe research team generated 3,600 exact rational number questions and expanded them into 8,600 prompts, covering five identity-preserving transformation families.
50%UnverifiedThe study evaluated five open-weight model systems.
50%UnverifiedThe five systems achieved standard answer accuracy ranging from 0.969 to 0.996.
50%UnverifiedAfter introducing representational transformations, orbit accuracy dropped to 0.848–0.981 and orbit invariance dropped to 0.851–0.981.
50%UnverifiedMost of the widespread failures produced by the strict parser do not stem from model reasoning errors, but rather because multiplicative-form scientific notation falls outside the implemented number grammar.
50%UnverifiedFlaws in the evaluator interface can masquerade as reasoning failures, and failing to distinguish between the evaluation pipeline and model capability when interpreting benchmark results may lead to misleading conclusions.
50%UnverifiedThe study is accompanied by a companion archive containing a frozen benchmark test set, evaluation audit logs, raw responses from the consensus experiments, and analysis code.
50%