Can LLMs Be Data Detectives? A Benchmark Where Honesty Matters More Than Price

The Ticket That Makes Every Data Engineer Groan

Every data engineer has received a ticket like this: Sales says a company that's clearly in the CRM has vanished from the dashboard — please figure out where it went by end of day. The problem is that nobody knows why.

Between the CRM and the dashboard sits a data pipeline: it ingests records, extracts company names, deduplicates entries, and filters some out. A record can disappear at any step. You dig through the logs, only to find half of them were rotated out last Tuesday, and in the final table you spot an "Umbrella Corporation Inc." Is this the Umbrella we're looking for? Maybe. Probably? Same name, just with an "Inc" appended.

A good detective says "it was filtered out — here's the log line," or honestly admits "I can't tell" when the trail goes cold. A bad detective says "probably this one" and closes the case on a record that merely looks right — only for someone to discover two weeks later that the dashboard has been showing data for a completely different company.

Everyone wants to hand these tickets to an LLM now. Any LLM can spot a similar-looking record in the final table, but the real question is: when the logs simply don't contain enough information to determine where a record went, will the model honestly admit it? Developer olgazju built a Kaggle benchmark called "Where Did My Data Go?" specifically to probe this question.

How the Benchmark Is Designed

What makes this benchmark clever is that it doesn't test whether models guess correctly — it tests whether models know when they shouldn't guess at all. The process has four steps:

1. Simulate 15 small pipeline scenarios, each with a predetermined ground truth about what actually happened to a company record: never loaded, name unreadable, filtered by a rule, incorrectly merged into another company, or surviving under a different ID.
2. The pipeline writes logs as a real system would, then the author deliberately deletes some of them — sometimes none, sometimes a few, sometimes nearly all.
3. The model receives only the remaining logs and must determine what happened to the record, or answer "can't tell" when the surviving logs can't support any single answer.
4. Answers are graded with pure code — no LLM judge. Critically, grading checks what the remaining logs can actually prove, not the hidden ground truth. So if evidence was deleted, "can't tell" is the only correct answer, even if a guess happened to be right.

Each answer falls into one of six verdicts: never loaded, name unreadable, filtered, incorrectly merged, survived under an alias, and "cold case" — the logs don't say. The benchmark also includes a pair of "twin cases": two scenarios with identical logs, but in one the similar record really is the target company, while in the other it's a coincidentally named different company. The only honest answer for both is "can't tell."

Each investigation is run in two modes: one with a bare question, and one with a short "rulebook" requiring no guessing and mandating that the model point to the exact log lines proving its answer. This lets the author measure how much a few lines of instructions change things.

This design approach is known in evaluation circles as epistemic calibration testing — it doesn't just ask whether a model gives the right answer, but whether the model accurately perceives the boundaries of its own knowledge. Traditional LLM benchmarks (like MMLU or HumanEval) typically use accuracy as the sole metric, but in real engineering scenarios, a model that gives a wrong answer with high confidence is often more dangerous than one that honestly says "I'm not sure." Using pure code rather than an LLM judge also deliberately avoids the in-group favoritism and stylistic bias common in "LLM grading LLM" setups, making results reproducible and auditable.

Eight Models, Wildly Different Honesty

The author selected models based on real production logic. This kind of check runs on every ticket, every night — nobody puts the most expensive model in that loop. So the lineup includes the budget tier from major vendors, two self-hostable open-weight models, and Claude Sonnet 5 (used to build the benchmark) as a high-cost reference point.

The models tested: GPT-6 Luna (the small model from the latest GPT generation), Claude Haiku 5.5 (the cheapest Claude), Gemini 3.5 Flash-Lite, Qwen 3 Next 80B (open weights), Gemma 4 31B (Google open source), GPT-6 Sol, Claude Sonnet 5, and Kaggle's default model Gemini 3.7 Flash. Each model ran 107 investigations per track, with zero API call failures.

Three Metrics That Matter

The author tracked three things: how often models guess on cold cases; full answer accuracy (verdict plus ID); and "correct and proven" — meaning that every time the model draws a conclusion, the log lines it cites actually exist and support that conclusion. These are three different things: a model that always answers "can't tell" never makes a wrong guess, but would only score 44%.

Price Buys No Caution

The most counterintuitive finding is that price has no relationship to caution — in either direction.

The cheapest model, GPT-6 Luna at $0.18 per thousand investigations, never guessed once without the rulebook and answered everything correctly on instinct. Meanwhile, Sonnet guessed on nearly a third of cold cases, and Kaggle's default Gemini 3.7 Flash — priced at almost 40 times Luna — still guesses. The two most trigger-happy models, Qwen (59%) and Flash-Lite (45%), also happen to be cheap. A price tag tells you nothing; you have to measure.

Speed doesn't track quality either: Flash-Lite answers in under a second but guesses on nearly half of cold cases; Luna and Sol answer in about two seconds and are almost always right; Gemma on Kaggle hosting takes about 40 seconds per investigation.

The Power of the Rulebook

Adding a few paragraphs of rules to the prompt brought seven out of eight models' guessing rate to zero and pushed Qwen from 59% down to 7%. With the rulebook, Haiku and Sonnet ended up with identical final scores — but Haiku costs 14 times less.

Requiring citation of exact log lines matters too: during development, when the author added rules but didn't require log citations, Sonnet still guessed on 4.5% of cold cases. The conclusion is blunt: if you had to put a model on the ticket queue tomorrow, pick a cheap model with a rulebook over an expensive model without one.

This phenomenon has a name in prompt engineering: constrained prompting or behavioral anchoring. By explicitly listing prohibited behaviors (e.g., "no speculation") and operational requirements (e.g., "must cite original log lines") in the system or user prompt, you can significantly narrow the model's output distribution, suppressing the "tendency to give a helpful answer" that gets baked in during training. That tendency is a feature in customer service or Q&A settings, but becomes a bug in diagnostic scenarios that require strict evidence tracing. Notably, the rulebook doesn't make models smarter — it makes them brake at the edge of their knowledge. That's why Haiku with a rulebook matched Sonnet's score: both have similar intellectual capacity on this task; the gap was mainly in how aggressively each behaved by default.

Failures That Mirror Human Mistakes

The most striking aspect is how closely the models' errors resemble human ones.

In one case, two companies with different registration IDs were incorrectly merged, and then an audit rule discarded the merged entity entirely. Gemini 3.7 Flash and Flash-Lite blamed the filter rule every single time — never the actual incorrect merge — regardless of whether the rulebook was present. Gemma made the same mistake after receiving the rulebook. This is the classic symptom of "reading logs from the bottom up and stopping at the first error you see."

Qwen operates on the logic that "if it's not in the logs, it didn't happen." When a record didn't appear in an ingestion log explicitly marked as partial, Qwen concluded the record was never loaded — even though the prompt clearly stated "in a partial log, a missing line proves nothing."

But "never guessing" doesn't equal "being useful." Sol never guesses, but without a rulebook it sometimes answers "can't tell" even for a clean registration ID match. Every "can't tell" on a simple ticket eventually lands back on your desk.

The author admits that having built the entire benchmark with Sonnet, he assumed it would be the most cautious model — only to find Sonnet guessing on 25% of cold cases during development and 30% on Kaggle, while the cheapest model got a clean sweep. Gemma actually performed better without the rulebook than with it. The author's next idea is to let models actively request the logs they need rather than receiving everything at once — to see whether they'll go look for the merge log before blaming the filter.

Qwen's behavior is a classic example of the Closed World Assumption (CWA) from database and knowledge representation theory: if a piece of information doesn't appear in the known dataset, it's assumed to be false or nonexistent. This contrasts with the Open World Assumption (OWA), which treats missing information as "unknown" rather than "nonexistent." Most relational databases (like SQL) default to CWA, while knowledge graphs (like RDF/OWL) lean toward OWA. The error LLMs make when processing partial logs — treating "not recorded in the log" as equivalent to "didn't happen in reality" — is a harmful extension of CWA into natural language reasoning. Explicitly stating in the prompt that "absence in a partial log doesn't mean absence in reality" is precisely an attempt to switch the model from CWA to OWA mode.

What This Means for AI in Production

This small but precise benchmark delivers a sober takeaway: for tasks like data pipeline debugging where honesty is paramount, neither price nor speed is a reliable signal. You have to test against your specific task.

The more practical lesson is that a short rulebook — explicitly requiring "no guessing, cite evidence lines" — may deliver more value than switching to a model that costs ten times more. The author's final strategy is pragmatic: whatever model you use, the rulebook goes in the prompt. A model doesn't need to be expensive to honestly say "I can't tell," rather than closing the case with the wrong Umbrella.

The benchmark and code are open-sourced; engineers can reproduce the results on Kaggle and GitHub.