We Tested 12 AI Models on Prediction Markets — None Beat the Market

When AI Meets Prediction Markets: An Experiment With No Winners

Prediction markets have long been seen as a touchstone for testing collective intelligence. From election outcomes to sports events, market prices are often more accurate than any single expert's judgment. So what happens if we let the 12 most advanced AI models of today play this game — can they beat the market?

A Reddit user (marked as original content in the post) ran a systematic backtest: putting 12 AI models against historical prediction market data, letting them make judgments on events that had already occurred, and then comparing them with the market prices at the time. The conclusion was very direct — not a single model could consistently beat the market.

This result isn't surprising, but the evidence itself deserves serious attention. It reveals a fact obscured by the AI hype: in domains where information aggregation is extremely efficient, the advantages of large language models aren't as large as imagined.

Experimental Design: An "Open-Book Exam" Using Historical Data

According to the original post, the core idea of the experiment was to present models with prediction market questions that had already been settled, each with a clear price in the market (usually expressed as a probability). The models needed to give their own probability estimates, and then the researchers compared the models' judgments with market prices to assess which was closer to the final outcome.

There's a key design element here: the models were facing historical events, so in theory they might have already "seen" the answers to these events. Even so, the 12 models as a whole still did not demonstrate predictive ability beyond the market. This is particularly worth pondering — if they can't beat the market even on an open-book exam, their performance in closed-book scenarios would only be weaker.

The models involved in the experiment covered different vendors and sizes, but the original post did not disclose the specific list and score details one by one. What can be confirmed is that the sample size was 12 models, the evaluation target was historical prediction markets, and the criterion for winning or losing was prediction accuracy relative to market prices.

Why the Market Is So Hard to Beat

Prediction market prices are essentially the result of participants voting with real money. They aggregate dispersed information, different judgments, and continuously updated expectations. The price of an efficient market already compresses a large amount of private information.

The problem with AI models is that their judgments rely on statistical patterns in training data. When faced with tasks that require integrating the latest information, handling niche events, or identifying market mispricing, models often lack independent information sources. They are more like restating the "consensus" rather than discovering "deviations."

Another angle is calibration. Market price calibration is usually refined through repeated corrections by a large number of traders, while model probability outputs tend toward overconfidence or conservatism. Even if a model judges certain individual items correctly, overall calibration error will drag down final performance.

Prediction Market is a market mechanism that lets participants trade contracts on event outcomes using real money. The contract price directly reflects the market's collective estimate of the probability of that event occurring — for example, a price of 0.7 means the market believes there is a 70% chance the event will happen. Its core advantage is that prices are continuously aggregated from dispersed information and judgments through trading, and any new information is quickly incorporated into prices by arbitrageurs. Therefore, prediction markets are often seen as an embodiment of "collective wisdom," and their calibration is often better than expert judgment or polls. The "market price" referred to in the article is precisely this probability estimate formed through continuous trading, representing the result of a large number of participants voting with their capital.

Calibration is a key indicator for evaluating the quality of probabilistic forecasts, referring to the degree of match between predicted probabilities and actual occurrence frequencies. A well-calibrated model, when it gives a 70% probability, should see the actual occurrence frequency of such events close to 70%. Market price calibration usually becomes accurate through repeated corrections by many traders, because mispricing leads to losses and is corrected by arbitrage. In contrast, large language models' probability outputs often have systematic biases, such as overconfidence (giving extreme probabilities but with insufficient actual accuracy) or conservatism (probabilities approaching 50%). Even if a model judges a single event correctly, overall calibration error can significantly drag down its performance relative to the market.

What This Means for AI Applications

This experiment serves as a reminder for AI practitioners and product managers: don't treat large language models as prediction machines. In highly marketized prediction scenarios such as finance, elections, and sports, models are better suited to auxiliary roles — such as organizing information, generating scenario analyses, and identifying overlooked variables — rather than directly giving "final answers."

For teams wanting to build prediction-type AI products, this result means they need to redesign their value proposition. If the product's core selling point is "more accurate than the market," then it will most likely fail. A more viable direction is to provide incremental information beyond the market, or to help users understand why the market is priced the way it is.

Limitations and Open Questions

The original post's experiment has some details that were not fully disclosed: prompt design, model temperature settings, whether multi-turn reasoning was allowed, how insufficient market liquidity was handled, and so on. All of these factors can affect the results. In addition, the overall conclusion for 12 models does not mean every model performed mediocrely — there may be individual models that approached or matched market levels in specific categories, with the overall average simply pulled down.

There's another question worth asking: if models were specifically fine-tuned to target market mispricing, would the results be different? There is no answer yet, but this is exactly a direction for follow-up research.

Conclusion: The Market Is Still the Strongest "Model"

12 AI models, zero wins. This result is less a rejection of AI than a reaffirmation of market mechanisms. The efficiency of prediction markets in aggregating information and calibrating probabilities still exceeds the capability boundaries of current general-purpose AI models.

For AI researchers, this is a clear baseline: if you want to prove a model has predictive ability, first beat the market. For ordinary users, when seeing "AI prediction" products, a bit of skepticism is reasonable.