Same Question, Different Answer: Where Does AI Go Wrong?

In the world of payroll software, "close enough" is a polite way of saying wrong. A paycheck off by one krona still triggers a correction, an explanation, a signature. A payslip sent three years ago must still comply with the calculation rules in force at the time.

So when someone says "I don't trust AI because the same question gets a different answer every time," I mostly agree. But what I really want to know is: why do the answers differ. So I went and asked the party that gives the answers.

"If I ask you the same question twice, will I get the same answer?"

Not necessarily. But the phrase "the same question" is carrying far more weight than it should.

"What do you mean?"

I mean that most questions leave room. Which stock, which price, which time period. Every time, I have to fill in those gaps — and I might fill them in differently.

I know this pattern all too well from payroll. "Salary" can mean base pay, gross pay, net pay, including allowances, including quarterly bonus. Ask five people for the same employee's salary and you might get five correct answers, corresponding to five different questions. That isn't an unreliable system; that's an imprecise question.

But what really keeps me up at night isn't the different answers.

"What worries me is the kind of wrong answer nobody catches. A filter built slightly off. An employee list that's quietly incomplete. No error message. Just someone who didn't get paid."

Then let's go find where that kind of error is actually born. There's more than one place.

Birthplace One: You Asked in the Wrong Place

The first birthplace of errors exists before any answer appears at all.

A few days ago I wanted to try a new MCP server, so I typed into the chat box: Use canli-mcp to find out who owns more than 5% of Coca-Cola? Nothing useful came back. That server runs on your own computer, through the desktop app. I was asking in a browser, which can never reach it.

I can't use a tool that isn't connected to that conversation, and this time I said so. The risk is the other case: a question that sounds answerable, plus a model that answers anyway — from memory.

"From memory means..."

From what I learned during training. Which could be months old. I know Coca-Cola's long-standing largest shareholder. I don't know last week's filing.

This is exactly where my story with AI began. I asked for Amazon's stock price, got a polite refusal, and thought: that's it? Only later did I understand the question was asked in the wrong place. Without a live data connection, the honest answer is "I can't." The dangerous answer is a confident, stale number.

MCP (Model Context Protocol) is a standard open-sourced by Anthropic in November 2024 for connecting large language models to external tools and data sources. Think of it as the "USB-C port for AI": as long as a server follows the protocol, any compatible AI client can call it.

An MCP server is a standalone program that can run locally on a desktop or be hosted remotely. The client (such as a chat app) must first establish a connection with that server before the model can call the tools it exposes. The canli-mcp mentioned here is exactly such a financial data server — it runs on the author's own computer, so it is only recognized in the desktop app; in a browser-based conversation, that connection simply doesn't exist, and the model can't reach it.

This matters because what a model "can answer" depends not only on its own knowledge but also on which tools are actually mounted in that particular conversation. When the connection is missing, the safest response is to clearly say "I can't access that tool," and the most dangerous is for the model to fall back on its training memory and produce a plausible-sounding but potentially outdated number.

Birthplace Two: The Model's Own Choices

"So what about when the tool is properly connected?"

Then the data is the same every time. What can vary is what I request from the tool. Which ticker symbol. Close or adjusted close. Which dates.

And here we're back to the salary example. The data in my payroll system doesn't change between two reports. The report changes, because someone chose gross over net.

Tools make choices too. Developer Arhan Canli has publicly written about a bug in an early version of canli-mcp (the free, MIT-licensed financial server he built). Ask it for "the price of silver" and it returned US consumer inflation data. The word "price" matched a description in the inflation series; "silver," the actually important word, matched nothing. The tool didn't crash. It returned real numbers, neatly formatted, answering a question nobody had asked.

That's the kind of error that worries you. Not an error message, but the right answer to the wrong question.

His fix was simple and, I think, wise: when the tool isn't sure, it now says "I don't know" and points elsewhere. My old programs worked like that by default. When something was missing they crashed — loudly, rudely, safely.

"Close" and "adjusted close" are two easily confused but semantically distinct fields in financial data. Close is the actual traded price at the end of a given day's session; adjusted close retroactively corrects for corporate actions like splits and dividends, making historical prices comparable to current ones.

For example, if a stock does a 2-for-1 split, the actual close before the split gets "halved" to keep the series continuous. If you calculate long-term returns using unadjusted prices, you'll reach the wrong conclusion. The "raw" pre-split closing price of $498.3201 for Tesla in 2022 in this article is exactly three times its split-adjusted price — meaning that number was reconstructed as an adjusted price, not the actual price traded that day.

So these seemingly trivial gaps — which stock, which price, which time period — actually determine the semantics of the entire calculation. The different choices a model or tool makes in filling those gaps can direct the same question toward completely different results, while the numbers themselves remain perfectly formatted and apparently correct.

Birthplace Three: The Silent Filter

"So where does the quiet kind of error come from?"

Sometimes from the tool itself. Let me show you one we made together.

Last week, for a story about congressional stock trading, we pulled all trades from the last week of July and measured how long each one took from trade to disclosure. A hundred trades. Clean numbers, a tidy median, an honest note about the small sample size. Published.

Days later, while checking reader questions, one field in the server's response said something the table never had: total: 111.

A single request returns at most a hundred rows. The other eleven were there all along. No failure, no warning. The data just ended.

"That's my incomplete employee list."

Yes. Every number we reported was correct — for the hundred rows we saw. The error was assuming those hundred rows were all of them.

I wrote payroll filters for years, and I know this failure all too well. It never looks like an error. It looks like a shorter list. The only defense I trust is to ask a second, boring question: how many should there be? Compare the count to the list, and the silence starts to speak. With the data we were using, that question cost nothing — the total was right there in the response, waiting to be read.

Does the Server Choice Really Matter?

"If tools can hide rows, how much does it really matter which tool you choose?"

Someone measured it. Arhan Canli — the developer with the silver bug — used one model, one prompt, the same constraints, and changed only the server behind it. On questions every server could complete, canli-mcp got 22 right out of 28 runs. The best alternative, EdgarTools plus Yahoo, got 15 out of 28.

This is the first measurement I've seen where everything but the server is held fixed. It confirms my suspicion: the same model, connected to different things, can be more or less accurate.

He raised a caveat himself: each question was asked twice, and an easy question is often right both times. So it's closer to fourteen questions than twenty-eight. Promising, but unproven. He's running a new set of questions, written before any tuning, and has promised to publish the results regardless of how they turn out.

I like that last part more than the numbers.

EdgarTools is an open-source Python library for scraping listed-company filings from the SEC's EDGAR system; Yahoo (usually referring to interfaces like yfinance) provides market data. Combined, the two could in theory answer questions similar to those canli-mcp covers, which makes them a reasonable control group.

The key design point of this comparison is "controlling variables": the same model, the same prompt, the same constraints, with only the server behind it changed. Only this kind of design can genuinely separate "how strong the model is" from "how reliable the data pipeline is," rather than letting readers attribute everything vaguely to "whether AI works."

The sample-size limitation is worth noting. If each question was asked twice across 28 runs, the actual number of distinct questions covered is only about 14, so a single right or wrong result can heavily perturb the conclusion. That's why the author values the developer's commitment to "write the questions before tuning and publish the results no matter what" — a pre-registered test says far more than picking flattering numbers afterward.

Birthplace Four: The Source Itself Is Wrong

"What if the model asked the right question and the tool returned every row?"

Then there's one last place: the data itself.

Two examples, both found by others.

In an article about stock splits, I printed Tesla's "raw" pre-split closing price from 2022: $498.3201. A reader — again, Arhan Canli — noticed the fourth decimal. Stocks don't trade in hundredths of a cent. That number was exactly three times Tesla's split-adjusted price. It wasn't a price someone actually paid; it was a price someone reconstructed. Of the 3,207 Tesla closes in my sample, 1,449 had more decimal places than any exchange would ever report.

That means some checks were comparing the data to itself. It passed — because it couldn't fail.

A second example also comes from him. The SEC publishes the exact acceptance time of every filing, and its submissions API keeps one file per company. He measured 450 filings from 25 companies. In twelve of the files, the times were right. In the other thirteen — including Apple, Nvidia, and Amazon — every time in the file was off by exactly one New York time-zone offset: four hours in summer, five in winter. A JPMorgan earnings filing accepted at 6:30 a.m. New York time appeared to have been accepted after the market opened. The same filing showing one time in company A's file and another time in company B's file. Same filing, different file, different time.

Neither error announces itself. Both have a cheap check: count the decimal places; compare one file's time against a second source.

The SEC's submissions API maintains a structured file for each listed company, recording metadata on its historical filings, including an acceptance datetime precise to the minute. The problem is that these time fields don't always carry a time-zone label, or the labeling isn't consistent across different files.

When the times in a given file are uniformly shifted by exactly one New York time-zone offset (four hours in summer, five in winter), you get the phenomenon of "the same JPMorgan filing showing one time in company A's file and another time in company B's file." On the surface the data is complete and neatly formatted, with no error, but the semantics are already wrong.

The danger of this kind of error is that it can't be caught by internal validation: check this data against itself and it will always "pass." The only cheap and reliable check is to bring in a second, independent source for cross-comparison. This also confirms the article's main thread — errors aren't necessarily born in the model; they may have been waiting at the source all along, and what people need to do is design the contrastive check that exposes them.

Final Thoughts

"Same question, different answer" ultimately decomposes into four distinct problems. The question was asked in a place the data couldn't reach. The model or tool filled the question's gaps differently. The tool returned less than everything, and said nothing. Or the source was already wrong before anyone asked.

None of these are solved by trusting AI more or less. Each is solved by a cheap, boring check: is the tool actually connected? What exactly did it query? How many rows should there be? Does this number look real?

That's also why I agree with the people around me who say AI shouldn't calculate payroll. It shouldn't. A calculation repeated for every employee every month belongs in a program that has been verified once and behaves the same every time. Prompts are for questions you'll only ask once, for helping you find the data first. The final check, and with it the responsibility, always belongs to a human.

My challenge still stands: find me a task where the check costs more than the generation, and where the prompt is still the right choice. I haven't found one yet. But I have learned where to look first when an answer seems off: not the model, but the chain behind it.

One-line Summary

"Same question, different answer" is really four independent types of error, defended against by cheap second checks, with responsibility ultimately resting on humans.

Paragraph Summary

The article breaks the vague complaint that "AI gives a different answer every time you ask the same question" into four mutually independent birthplaces of error: first, the question was asked in a place the data couldn't reach (e.g., the tool wasn't connected, so the model answered from memory); second, the model or tool filled the question's gaps differently (e.g., wrong ticker symbol or price convention); third, the tool silently truncated the data, returning a hundred-row result that looked complete but was part of a "total: 111"; fourth, the data source itself was wrong, such as reconstructed split-adjusted prices or SEC acceptance times that differ by a time zone across files. Drawing on the strict standards of payroll software, the author stresses that none of these are solved by "trusting" or "distrusting" AI more, but by a few cheap, boring second checks — is the tool connected, what exactly did it query, how many rows should there be, does the number look real. The final check and the responsibility always belong to a human.