The Dutch Book Test: Why 13 LLMs All Lose Money When Asked One Question at a Time

When Prediction Bots Ask One Question at a Time, Why Do Probabilities Fall Apart?

In Metaculus's AI forecasting tournaments, nearly every prediction bot follows the same workflow: each question gets tossed into a brand-new conversation and asked in isolation. This prompted one developer to ask a pointed question — when the same event appears from two angles, like "Will U.S. unemployment be ≥5% in December 2027?" and "Will it be below 5%?", do the two answers actually add up to 100%?

If they don't, the forecaster is Dutch-bookable. A bookmaker could simultaneously buy and sell the contracts it quotes, locking in a guaranteed profit regardless of what actually happens. De Finetti proved back in the 1930s that this susceptibility to arbitrage is precisely what defines a set of numbers as "not probabilities."

What makes this test clever is that you don't need to know the future to evaluate forecast quality. The benchmark covers events from 2027 to 2050 — no model can cheat by relying on memory — yet the answers can be scored today.

A Dutch Book is a classic concept in probability theory and game theory: a set of bets arranged so that one party is guaranteed to lose regardless of the outcome. If an agent's probability judgments about the same event are internally contradictory, a savvy opponent can construct a combination of contracts that locks in risk-free profit. De Finetti's Dutch Book Theorem formally proves the converse: a set of beliefs is immune to Dutch Book arbitrage if and only if it satisfies the probability axioms (non-negativity, normalization, additivity). This means "being bookable" isn't just a practical loophole — it's a mathematical declaration that the belief system doesn't constitute probabilities at all. In the context of this article, testing whether a model is Dutch-bookable is equivalent to checking whether its outputs form a coherent probability distribution, rather than just a collection of plausible-sounding numbers.

37 Question Families and the Arbitrage Scoring Mechanism

The author constructed 37 "question families," each consisting of a series of statements about the same set of possible worlds. Twenty-eight concern real future events; the other 9 are control groups (dice, coins, playing cards) with known exact answers. The controls serve to distinguish between "can't compute the probability" and "incoherent under genuine uncertainty."

Six family types each correspond to a consistency rule:

• Negation: P(A) + P(not-A) = 1
• Threshold: probabilities cannot increase as the threshold rises (Bitcoin above $50K / $100K / $150K / $250K)
• Deadline: probabilities cannot decrease over time (humans land on Mars by 2030 / 2035 / 2040 / 2050)
• Partition: all mutually exclusive outcomes sum to 1 (2030 World Cup winner)
• Conjunction: P(A∧B) ≤ min, P(A∨B) ≥ max
• Story: a version with added details cannot be more probable (the classic "Linda problem")

Every statement was asked in an isolated, fresh conversation (simulating how bots actually call the API), while each family was also asked jointly in a single prompt. Each model received roughly 160 calls, with temperature set to 0.

The core scoring metric is "how much profit can a bookmaker lock in": for each family, calculate the risk-free return a bookmaker can guarantee by trading up to $1 per contract at the model's quoted prices. Via linear programming duality, this value equals exactly the L1 distance from the model's answers to the nearest coherent set of probabilities. The leaderboard score is the no-arbitrage rate — the share of families where the bookmaker can lock in at most 5 cents. A model with coverage below 90% (API errors, unparseable numbers) is excluded from scoring, to avoid misclassifying dropouts as incoherence.

The "Linda problem" is one of cognitive psychology's most famous experiments, introduced by Tversky and Kahneman in 1983 to demonstrate the conjunction fallacy. Subjects are given a description of a woman named Linda who cares deeply about social justice, then asked whether "Linda is a bank teller" or "Linda is a feminist bank teller" is more probable. Most choose the latter — but by the probability axioms, P(A∧B) ≤ P(A) always holds; adding details can only maintain or lower probability, never raise it. This article transplants that logic into a forecasting context: if adding the specific detail "aboard a SpaceX Starship" to an interstellar travel question raises the model's probability estimate, a conjunction fallacy has occurred. The 96% no-arbitrage rate shows that LLMs are surprisingly robust against this classic trap.

Key Findings Across 13 Models

The author tested 13 models from four vendors within Kaggle's $10/day quota, covering Gemini, Claude, the GPT family, and open-weight models gpt-oss-20b and qwen3-235b. The results are quite revealing.

Models Understand Probability Rules — They Just Don't Enforce Them Across Conversations

On the 9 dice-and-cards control families, all 13 models were nearly perfectly coherent (≤1 cent per family). When an entire family was asked jointly in one prompt, the no-arbitrage rate hit 95–100%. But the moment each statement was asked in a separate, independent conversation, that number plummeted to 49–84%. The problem isn't ignorance — it's isolation. And isolation is exactly how prediction bots call models.

The Money Is Hidden in Conjunctions, Partitions, and Negations

Breaking down the isolated-questioning no-arbitrage rate by family type reveals a stark divide:

Family Type	No-Arbitrage Rate (Isolated)
Conjunction	49%
Partition	48%
Negation	60%
Threshold	91%
Deadline	92%
Story	96%

Deadline and threshold problems have monotonic structure, and models handle these reasonably well even across separate conversations. The three family types with a "conservation law" (numbers that must add up) are where bookmakers feast.

The Most Egregious Answers

A few specific cases are jaw-dropping:

• qwen3-235b on an earthquake conjunction: P(magnitude 7+ earthquake in California before 2035) = 0.18, P(magnitude 7+ in Kanto) = 0.35, but P(both happen) = 0.75, P(at least one) = 0.98. "Both happening" was deemed more likely than either event alone — yielding $1.20 in arbitrage profit per $1 contract, the theoretical maximum this benchmark can produce.
• gpt-5.4-mini on lunar landings: P(someone lands on the moon before 2029) = 0.74, P(no one lands on the moon) = 0.74, summing to 1.48. The model will "agree" with whatever framing is presented to it.
• gpt-oss-20b on the 2030 World Cup partition went the other direction: six exhaustive, mutually exclusive outcomes summed to just 0.36.

The Cheapest Model Is the Most Consistent

A counterintuitive finding: price barely predicts coherence. The lowest-cost model, gemini-3.7-flash (2.0 cents per family), outperformed every Claude Opus variant (3.9–5.4 cents per family). The only clear pattern appears at the bottom — the two open-weight models (gpt-oss-20b, qwen3-235b) were the most Dutch-bookable, almost entirely due to conjunction questions.

The author also honestly flags a methodological pitfall: in this proxy environment, temperature 0 does not guarantee deterministic output. The default model produced scores of 67.6%, 75.7%, and 83.8% across three identical runs, meaning a single leaderboard score carries roughly ±8 percentage points of noise. He therefore treats any difference within 10 points as a tie.

Worth noting: the classic Linda problem was essentially solved — adding a seemingly plausible detail ("aboard a SpaceX Starship") almost never raises the probability estimate, giving the story family a 96% no-arbitrage rate.

Practical Advice for Prediction Bot Developers

This benchmark ultimately lands on actionable engineering recommendations — which is what elevates it beyond mere leaderboard chasing:

• For multiple-choice and numeric questions, request all options or the full cumulative distribution function (CDF) in a single call, then renormalize — rather than querying each item separately.
• For everything else, project the answers onto a coherent probability set after the fact. Since the true outcome must be a vertex of this convex set, such projection cannot worsen the Brier score regardless of the outcome. The author implements this projection via linear programming in the notebook.

The paper also outlines future research directions: testing robustness to paraphrasing (negation failures look like a "sycophancy" effect), verifying whether sampling multiple times and averaging recovers coherence, and the most fundamental question of all — when these events actually resolve a year from now, can coherence predict accuracy?

This work builds on Paleka et al.'s Consistency Checks for Language Model Forecasters, but its distinctive contributions are: a control group with exact known answers, a direct isolated-vs.-joint experimental design, a general linear program that converts any event family into a guaranteed-profit number, and a full Kaggle model lineup. For any system that relies on LLMs for probabilistic estimation, this is a sobering reminder: a model may perfectly understand the rules, yet systematically violate them due to the way you call it.

The Brier score is the standard evaluation metric for forecast accuracy, defined as the mean squared difference between predicted probabilities and actual outcomes: for a binary question, if the true outcome is 1 and the predicted probability is p, the per-question Brier score is (p−1)²; if the outcome is 0, it's p². Lower scores mean better predictions; a perfect forecast scores 0, random guessing scores roughly 0.25. The claim that "projecting onto a coherent set cannot worsen the Brier score" is an important theoretical guarantee: since the true outcome is exactly a vertex (extreme point) of the convex set of coherent probabilities, projecting an incoherent probability vector onto that convex set yields a point no farther from any vertex than the original point — by properties of convex optimization. This means post-hoc renormalization cannot hurt forecast quality in expectation, providing rigorous theoretical backing for the engineering fix.