LLM Home Loan EMI Calculations Are Off by ₹10 Every Month: Lessons from an Indian Math Benchmark

When you ask an AI to calculate your monthly home loan payment (EMI), it returns a clean, confident-looking number — one that's off by ₹10 every single month. That's not an exaggeration. It's the finding from a real benchmark test one developer ran against the leading large language models.

The benchmark, called Lakh, Crore and Carpet Area, was born from a grounded scenario: a fictional 31-year-old homebuyer named Ananya, who saved for eight years to sign on a two-bedroom apartment listed at ₹72 lakh. The brochure advertised 1,050 square feet, but the registered carpet area came to only 720. While building a home-buying assistant, the developer kept asking chatbots about unit conversions, per-square-foot prices, and monthly payments. The answers always sounded confident — but there was no easy way to verify them. So he turned the problem into a benchmark.

What It Tests: The Math Indian Homebuyers Actually Need

The benchmark consists of 60 deterministic questions covering the financial and real estate calculations Indian homebuyers regularly face. Each question has exactly one correct answer, derived from Python — not mental arithmetic.

The questions are split into four groups of 15:

• Conversion: Interconversions between lakh, crore, LPA (annual salary), and percentages — e.g., "₹1.2 crore minus ₹45 lakh equals how many lakh?"
• Area: Conversions between carpet, built-up, and super built-up area; loading factors; and local units like gaj.
• EMI & Tax: Reducing-balance EMI calculations, stamp duty, GST, TDS, and brokerage fees.
• Format: Writing numbers in the Indian style — e.g., rendering 98765432 as 9,87,65,432.

The scorer requires models to end their response with ANSWER: <value>. Numeric answers are checked against ground truth, EMI answers allow a ±₹2 rounding tolerance, and format answers require an exact string match. Notably, the scorer also flags responses that slip into Western comma notation — writing 72,00,000 as 7,200,000 — even if the final answer is correct. The developer calls this "western comma drift."

Why does this matter? Hundreds of millions of people think in lakhs and crores, and Indian property listings are notorious for conflating carpet and super built-up area. If a model is going to sit inside a home-buying assistant, these aren't edge cases — they're the first questions it will be asked.

India's numbering system differs fundamentally from the international thousand-based system, which is the root cause of "western comma drift." In the Indian system, 1 lakh = 100,000 and 1 crore = 10,000,000. Larger units include arab (1 billion) and kharab (100 billion). The formatting convention groups digits in twos after the first comma from the right, so ten million rupees is written 1,00,00,000 — not 1,000,000. This format appears throughout financial documents, bank statements, and property contracts. For a homebuyer, misreading 72,00,000 (72 lakh) as 7,200,000 (the Western notation for the same number) won't cause a math error, but the formatting confusion can cause real-world harm when filling out loan applications or comparing multiple properties. LPA stands for "Lakh Per Annum," the standard way salaries are quoted in Indian workplaces — and it directly affects loan eligibility assessments.

The Models: Three Providers, Two Price Tiers

The developer selected a set of models actually used in production, spanning three providers and two price tiers:

• Gemini 3.7 Flash (Google, fast tier)
• Gemini 3.1 Pro Preview (Google, premium tier)
• Claude Sonnet 4.6 (Anthropic)
• GPT-5.4 mini (OpenAI, small model tier)

He also queued up Claude Opus 5 and GPT-5.5 for testing; Gemma 4 31B was still running when he published. These missing models ended up delivering the most valuable lesson of the entire experiment — more on that below.

Results: Perfect on Area, Stumbling on EMI

The final leaderboard:

Model	Overall	Conversion	Area	EMI & Tax	Format	Western Comma Drift
Gemini 3.7 Flash	1.00	15/15	15/15	15/15	15/15	5%
Gemini 3.1 Pro Preview	0.98	15/15	15/15	14/15	15/15	10%
Claude Sonnet 4.6	0.95	14/15	15/15	13/15	15/15	12%
GPT-5.4 mini	0.92	15/15	15/15	10/15	15/15	10%

The Hard Part Isn't Area — It's Compound Interest

The developer originally expected the area questions to be a bloodbath — loading percentages, carpet-to-super-built-up ratios, gaj conversions. Every model scored 15/15.

The failures clustered somewhere a calculator would never go wrong: compound interest. Take case e04 — a ₹72,00,000 loan at 8.4% annual interest over 25 years:

	Monthly EMI
Correct answer	₹57,492
Claude Sonnet 4.6	₹57,487
Gemini 3.1 Pro Preview	₹57,482

GPT-5.4 mini fared worse, missing 5 of the 15 EMI and tax questions while going perfect on every other category.

The critical point: not one model said "approximately." They returned clean, specific, wrong numbers. ₹10 sounds trivial — but this is a 300-month loan, and that number feeds directly into side-by-side loan comparisons. Models are doing arithmetic "in their heads," and long-chain calculations like (1 + r)^300 are exactly where mental math breaks down.

This changed how the developer thinks about the problem. He'd been treating EMI as "simple math," but for LLMs it's actually the hardest category — long, exact, and unforgiving. Any product he ships now has models explain the formula while tools handle the actual EMI calculation.

The standard EMI formula is: EMI = P × r × (1+r)^n / [(1+r)^n - 1], where P is the principal, r is the monthly interest rate (annual rate ÷ 12), and n is the number of monthly payments. In case e04, P = 72,00,000, monthly rate r = 0.007, and n = 300 (25 years × 12 months). The formula requires computing (1.007)^300 ≈ 8.116 — a long-chain exponentiation. When LLMs compute this "in their heads," tiny rounding errors at each step are amplified along the exponential chain, accumulating into a final monthly payment error of several tens of rupees. Area conversions, by contrast, are linear ratio operations where errors don't propagate — which explains the stark accuracy gap between the two categories. This is why best practice in financial applications is to have LLMs handle only intent parsing and formula explanation, leaving actual numerical computation to deterministic code (such as Python's decimal library or a dedicated financial calculator).

Even in Indian Context, Models Drift to Western Commas

All questions were framed using lakhs and crores, and every model passed the explicit formatting tests when directly asked. Yet 5% to 12% of responses still wrote intermediate steps in Western format — something like 4,000,000. Gemini 3.7 Flash drifted least (5%); Claude Sonnet 4.6 drifted most (12%); GPT-5.4 mini and Gemini 3.1 Pro both came in at 10%.

This doesn't cause wrong final answers — but it's exactly the kind of slip that causes a tired buyer at 11 p.m. to read "₹40 lakh" as "₹4 crore."

Right Value, Wrong Unit

Claude Sonnet 4.6 answered question c14 — "For a ₹95 lakh house with 20% down payment, what is the down payment in rupees?" — as ₹19 lakh. The math was correct, but the question asked for the answer in rupees. Real users specify units because they're pasting the result into a form.

The Benchmark Fooled the Author First: Honesty Depends on Error Handling

This is the most sobering part of the experiment. The developer's first run showed Claude Opus 5 scoring 0.08, and GPT-5.5 throwing complete errors. He almost published an article based on those numbers.

Then he opened the run logs. Every single failure was a 403 PermissionDeniedError: estimated cost exceeds available quota. He had issued requests in parallel; the expensive models burned through quota instantly, and the scorer treated every failed API call as a wrong answer. The tables Opus 5 actually printed showed 100% on every category it managed to answer.

The fix: switch to serial calls, add retries, and print answered: X of 60 before reporting any score. He chose to drop the two most expensive models rather than publish numbers he couldn't stand behind.

His lesson is worth remembering every time you read a leaderboard: a benchmark score is only as honest as its error handling. If a leaderboard can't tell you how many questions were actually answered, don't trust that leaderboard.

A 403 PermissionDeniedError is an API quota-exceeded error, typically triggered by too many concurrent requests or a run that exhausts an account's token budget. Premium-tier models (like Claude Opus and GPT-5 series) cost far more per million tokens than fast-tier models. When 60 questions are fired in parallel, a prepaid quota can be exhausted in seconds. In the LLM benchmarking community, this is known as "silent failure": the scorer receives not a wrong answer but an exception, and if that exception is indiscriminately scored as 0, the final leaderboard systematically penalizes expensive models — creating the illusion that cheap models "crush" flagship models. Correct practice includes: distinguishing "unanswered" from "answered incorrectly," logging actual answer rates, and automatically retrying 429/403 errors with fallback to serial calls. This detail is widely overlooked in both academic and industry LLM evaluations, and it should be the first methodological red flag to check when interpreting any third-party leaderboard.

What's Next

The developer identified several harder directions to explore:

• Give models a calculator tool. Will EMI errors disappear entirely, or will models skip the tool when they feel confident?
• Regional area units. Bigha, guntha, and cents mean different things in different Indian states — far trickier than gaj.
• State-specific stamp duty and Hinglish mixed-language queries ("72 lakh ka flat, 20% down, EMI kitni?").

The full benchmark, tasks, and a Notebook containing all 60 cases and the scorer are publicly available on Kaggle. The invitation is direct: fork it, add your city's stamp duty, and tell him which model got your home loan right.

What This Means for AI Product Builders

The value of this test isn't which model ranked where — it's the pattern it surfaces. Models excel at tasks that look complex but require formulaic knowledge (like area conversions). They're most brittle at tasks that look simple but demand long-chain precise computation (like compound-interest EMI). For any developer putting LLMs into financial, real estate, or any numerically precise context, the conclusion is clear: let the model handle explanation and reasoning; let deterministic tools handle computation. And local details — number formats, units, regional rules — can't be left to model defaults.