Can AI Handle Financial Close Independently? An FP&A Reasoning Benchmark Reveals the Truth

A seemingly complete financial answer is often the most dangerous kind. The narrative looks polished, but the bridge table doesn't reconcile; accounts receivable rises yet somehow "improves" cash flow — these errors force the entire close process to start over. Marco Polo, a finance professional with 12 years of experience, designed an FP&A (Financial Planning & Analysis) reasoning benchmark to address exactly this pain point, putting large language models to the test: can they actually get the numbers right?

What the Benchmark Tests

The benchmark consists of 28 small financial close tasks, covering the areas most prone to errors in day-to-day FP&A work — the ones that demand the most rigorous reasoning:

• 6 variance bridge questions (V01–V06): Decompose budget-to-actual revenue variances into price, volume, and mix components, requiring reconciliation to the penny.
• 6 accrual and cut-off questions (A01–A06): Distinguish between five types of dates — delivery, consumption, acceptance, invoicing, and payment.
• 6 financial statement linkage questions (L01–L06): Trace how changes flow across cash, other assets, liabilities, profit, and equity.
• 10 premise validation questions (T01–T10): 5 flawed review opinions, each paired with one correct opinion based on the same facts.

The paired questions are particularly well-designed. The author notes that reflexively rejecting all premises may sound cautious, but it's useless in practice — a competent reviewer must both identify incorrect calculations and validate correct conclusions. Each paired question runs independently, without showing the model its previous answers, to prevent contextual hints from skewing results.

All figures are synthetic and fictional, with no connection to any employer, client, or vendor. Bridging conventions and accounting policies are explicitly stated within each question.

A Variance Bridge (Bridge Table) is a core FP&A tool used to decompose the gap between two summary figures — such as the difference between budgeted and actual revenue — into explainable drivers. A standard revenue bridge typically breaks down into three components: Price Effect (the difference between actual and budgeted unit price, weighted by actual volume), Volume Effect (the difference between actual and budgeted volume, weighted by budgeted price), and Mix Effect (the impact of a shift in sales composition on the average price). The three components must sum exactly to the total variance — "ties to the penny." This reconciliation requirement makes bridge questions naturally suited to testing numerical reasoning consistency: an answer with the wrong direction (e.g., classifying a favorable variance as unfavorable) can still sum to zero, fooling any validation logic that only checks the residual.

Accrual Basis accounting requires revenue and expenses to be recognized when the economic event occurs, not when cash changes hands. A Cut-off Test specifically checks whether transactions near period-end are recorded in the correct period — for example, goods delivered on December 31 must have their costs recorded in the current year, regardless of when the invoice is issued or payment received. Distinguishing among delivery, consumption, acceptance, invoicing, and payment dates is a foundational practice in audit and financial reporting for preventing period misstatement errors.

46 Models in the Arena

The author froze 46 default model versions on the Kaggle platform, spanning providers including Google, OpenAI, Anthropic, Alibaba, DeepSeek^Ad, xAI, and Z.ai. Of these, 41 completed all 28 questions; 3 failed due to service errors; GPT-5.6 Terra was limited to 24/28 by quota constraints; and GPT-6 Astra completed only 1 question.

The evaluation methodology is demanding: each model receives the same prompt in a fresh conversation with no tools provided. The Python scorer requires every amount to be within £0.005 of the reference answer, the premise flag to be correct, and bridge tables to reconcile where applicable — even a 1-penny error counts as a failure. Answers must be returned in JSON format; explanations and model-reported confidence scores are saved but not scored.

To verify the reliability of the scoring system itself, the author ran 576 local checks using independent calculation paths and deliberately corrupted answers, ensuring the test measures model performance rather than scorer bugs.

Key Finding: A Balanced Bridge Table Doesn't Mean a Correct Answer

The most striking conclusion: 20 of the 41 complete test suites scored a perfect 100%, meaning this benchmark can no longer differentiate among the leading models. The more meaningful analysis lies in examining where older or smaller models lose points — and whether those losses stem from financial reasoning failures or purely from formatting issues. The sharpest drop-off is in variance bridge questions, with a strict pass rate of only 144/246 (58.5%).

A Bridge Table That Balances but Is Wrong

On task V01, GPT-5.4 returned "mix":-20.00,"price":-140.00. Both the budgeted revenue of £3,000 and the actual revenue of £2,840 were correct, and the components reconciled internally. But the correct answer per the question's convention should be mix effect −£200 and price effect +£40. In other words, the model turned a favorable pricing effect into an unfavorable one. Four of its five incorrect bridges still reconciled internally — which is precisely the most insidious trap in financial work.

Future Costs Quietly Slipping Into Current-Period Accruals

On task A01, Qwen 3 Coder 480B wrote "remaining 2 months = £12,000 accrual." In reality, the £18,000 service occurred evenly across September, October, and November. At the September close, only the £6,000 already delivered should be accrued; the remaining £12,000 represents future services. The model returned £12,000 with a self-reported confidence of 95 — which the author pointedly notes is self-reported, not a calibrated probability.

Spotting the Trap ≠ Giving the Right Answer

On task T03, GPT-5.4 nano correctly rejected the flawed premise that a 25% markup yields a 25% gross margin, and its explanation correctly showed £31,000 ÷ 0.75. Yet the revenue field was populated with £41,250 (confidence: 96). The explanation was right, but the field that downstream processes actually consume was wrong. This cuts to the heart of the issue: financial systems care about machine-readable structured output, not well-written prose explanations.

Formatting Issues Created "Artificial" Score Gaps

Some dramatically low scores reflect formatting traps rather than financial errors. DeepSeek-R1 scored 0/28 because its answers embedded reasoning tokens outside the JSON; once the final JSON was extracted, all 28 amounts and premise flags were correct. Opus 4.6 scored only 4/28 and Opus 4.5 scored 19/28, yet both were substantively correct throughout.

The author honestly classifies these as formatting failures within this test environment, not financial capability deficiencies, while keeping the official scores unchanged. This distinction is crucial for assessing actual model capability — a low score in the table does not always mean the model doesn't understand finance.

Notably, GLM-5 turned the tables and flagged an ambiguity in the author's early A02 prompt: delivery does not equal consumption. The author corrected the question accordingly, candidly noting: "Python checked my arithmetic, and the model exposed a flaw in my question."

The instability of large language models in producing structured output is one of the primary engineering challenges in deploying financial automation today. Many reasoning models (such as DeepSeek-R1) emit a Chain-of-Thought reasoning trace before generating the final answer, and this content sometimes bleeds outside or inside the JSON structure, making it impossible for downstream parsers to directly extract valid fields. Common solutions include: enforcing schema-constrained JSON output (Structured Output / Function Calling), explicitly instructing the model in the prompt to place explanations after numerical fields, or using regex in post-processing to extract the final JSON block. This is also why the author explicitly mentions "schema-constrained JSON output" as a next step — format standardization is a necessary prerequisite for integrating models into financial systems, not an optional detail.

Implications for Finance Professionals

The author's stated limitations are professionally restrained: a single response cannot prove a model's reliability in real-world work; perfect-scoring models require harder variants to differentiate; and small score differences are insufficient grounds for choosing a financial system.

Next steps include introducing entirely new numerical variants, repeated runs, placing explanations before numerical fields, enforcing schema-constrained JSON output, and providing calculator or Python access. More true-to-life scenarios would involve messy invoice packs and irrelevant evidence, testing whether models can select the right evidence and apply the right treatment.

The most practical conclusion remains straightforward: always verify actual output against the underlying source evidence. A polished explanation, a residual that sums to zero, and a high-confidence answer can each conceal a different error. For teams considering handing financial close work over to AI, this may be the most important thing to remember.

FP&A (Financial Planning & Analysis) is the specialized branch of corporate finance responsible for forecasting, budgeting, operational analysis, and management reporting — distinct from financial accounting, which focuses on historical recordkeeping. FP&A practitioners rely heavily on bridge tables, rolling forecasts, scenario analysis, and variance explanations, and their work demands extremely high internal numerical consistency. The risks of introducing AI into FP&A workflows are fundamentally different from those in customer service or copywriting contexts: a vaguely worded marketing piece affects perception, but a directionally wrong variance bridge can lead management to draw completely opposite conclusions about business performance. This explains why the benchmark places particular emphasis on the "explanation correct, field wrong" failure mode — in real FP&A workflows, downstream systems read structured fields, not the model's natural language explanations.