Fixing LLM Arithmetic in Clinical Settings: A Deterministic Solver Approach

LLMs' Arithmetic Weakness Gets Amplified in Clinical Settings

Large language models excel at text comprehension but consistently stumble on basic arithmetic. In most applications, a misplaced decimal point is a minor nuisance — but in clinical calculator scenarios, a single wrong number can alter an entire medication recommendation or diagnostic conclusion. That's a patient safety issue with zero tolerance for error.

A study published on arXiv (arXiv:2609.10728) takes this problem head-on. The paper notes that the industry's typical response to unreliable model arithmetic is to hand-code each clinical calculator as a separately validated, fixed function. Precise as this approach is, it doesn't scale — every new calculator requires manual development and verification.

Program-Solve: Let the Model Write Code, Not Do the Math

The research team proposes an alternative: the model is not responsible for computation. Instead, for a given patient case, the model writes Python code that is handed off to a restricted local executor acting as a deterministic solver. This reduces the model's job from "compute the correct answer" to "decide how to invoke the solver."

The interface, called Program-Solve, is built around a separation of concerns: the model handles semantic tasks it's good at — reading clinical notes, identifying variables, selecting formulas — while offloading deterministic numerical computation to a genuinely reliable code execution environment. This follows the same logic as recent "tool use" and "code interpreter" approaches, but applies it specifically to a domain where numerical precision is non-negotiable.

Tool Use and Code Interpreter are two mainstream approaches to improving LLM reliability. The former lets a model dynamically call external APIs or functions during reasoning; the latter allows the model to generate executable code, receive the output, and continue reasoning from there. Both stem from the same observation: language models are strong at semantic understanding and logical planning, but weak at symbolic computation requiring precise state tracking. A restricted local executor goes a step further than a general-purpose code interpreter — it only allows a specific, sandboxed set of operations, preventing arbitrary system calls that would create security risks. This is especially critical in compliance-sensitive environments like healthcare. At its core, Program-Solve narrows the model's role from "end-to-end calculator" to "plan generator," using the executor's determinism to compensate for the model's unreliable floating-point arithmetic.

Rigorous Evaluation on MedCalc-Bench

The study evaluates on MedCalc-Bench Verified, a benchmark containing 1,100 patient cases across 55 calculators. Baselines include direct model arithmetic and a hand-coded library of 22 calculators. The models tested are Qwen2.5-7B and Qwen2.5-32B-AWQ.

Notably, the team didn't take the benchmark at face value. They audited every formula against current clinical guidelines and flagged 16 of the 55 calculators for issues related to versioning, intended use, or coefficient discrepancies. This kind of critical scrutiny of evaluation data itself is refreshingly rare in an era where benchmark-chasing is the norm.

Model Scale Determines the Solver's Value

When formulas and standard variables were provided, and both approaches had full access to the patient record, results showed a clear dependence on model size:

• 7B model: Using the solver is not a reliable advantage. Accuracy of 75.31% vs. 72.02% for direct arithmetic — a paired improvement of +3.29 percentage points — but the 95% calculator-clustered confidence interval spans zero at [-3.49, 10.38], meaning the advantage is not statistically significant.
• 32B model: The solver delivers a clear benefit. Accuracy of 90.53% vs. 83.47%, a +7.05 percentage point improvement, with the interval [0.47, 14.60] firmly above zero.

In short, adding an executor doesn't help all open-source models equally — larger models are better positioned to leverage this deterministic computation mechanism.

MedCalc-Bench is a benchmark dataset designed specifically to evaluate LLM performance on clinical calculation tasks, covering tools like eGFR for kidney function, CHA₂DS₂-VASc for cardiovascular risk, and fluid replacement calculations. The Verified subset has been manually reviewed to remove cases with ambiguous formula descriptions or incomplete patient records, ensuring reproducible evaluation results. It's worth noting that clinical calculators are not monolithic — the same scoring system (e.g., Glasgow Coma Scale) may have coefficient differences across different versions of clinical guidelines, and such differences are easily overlooked in automated evaluation. The team's manual audit of 16 problematic entries directly addresses this blind spot.

The paired improvement and confidence interval framing comes from a paired statistical testing framework: researchers run both methods on the same set of cases, then compare per-calculator paired differences, using clustered confidence intervals to correct for within-calculator sample correlation. This is more rigorous than comparing overall accuracy directly, since different calculators vary widely in difficulty. For the 7B model, an interval spanning zero ([-3.49, 10.38]) means we cannot statistically rule out the possibility that the executor provides no benefit — or even hurts performance. For the 32B model, a positive lower bound ([0.47, 14.60]) provides a stronger statistical guarantee.

The Precision and Limits of the Hand-Coded Library

As a point of comparison, the hand-coded library achieved perfect precision on the 440 cases it covers — but chose to abstain on the rest, yielding an overall coverage rate of just 40.0%. This cleanly illustrates the classic trade-off: either precise but narrow, or broad but imprecise.

Program-Solve's value lies in attempting to break this trade-off — using the model's generalization ability to achieve broader coverage while using the deterministic executor to preserve a floor on computational accuracy.

A Grounded Conclusion: The Executor Is Not a Silver Bullet

The paper's conclusions are notably restrained. Even under fully aligned conditions — matching formulas, variables, and record access — adding an executor helps some open-source models more than others. And it neither replaces validated formulas nor reliable variable extraction.

This point deserves emphasis: Program-Solve addresses whether the math is done correctly, but the clinical calculation pipeline has two equally critical steps upstream — whether the right formula was selected, and whether the variables extracted from the clinical record are correct. The deterministic solver fills in the last piece of the puzzle; it doesn't fix everything else.

For teams exploring LLM deployment in high-precision domains like healthcare or finance, this work offers a useful architectural pattern — and a clear reminder that model scale, formula validation, and variable extraction all matter, and none can be skipped.