Three AI Decision Models vs. Polymarket: Which One Can Best Predict the Future?

OpenAI recently released its Decisions API in beta, and instead of running traditional benchmarks, one developer designed a far more interesting experiment: pit three leading decision models against each other to predict future events, then compare their outputs against real prediction market odds.

This open-source project puts OpenAI's Decisions API, TypeSafe's Jev, and Cloudflare's Clef (open weights) under the same set of evidence, observing how they reason to vastly different conclusions from identical news — then benchmarks those conclusions against live prices on Polymarket and Kalshi.

Why Use Prediction Markets as the Judge

The developer's reasoning is straightforward: benchmarks are broken. Almost every model may have seen the standard answers during training, so leaderboard scores don't reflect genuine reasoning ability.

Predicting the future, on the other hand, has no answer key — no one's training data contains events that haven't happened yet. More importantly, prediction markets provide an objective reference point. As the author puts it: "Polymarket and Kalshi prices aren't based on gut feelings — real money is on the line." When three AI models read the same news and reach conclusions that contradict millions of dollars in real capital, one side will inevitably be proven wrong by the market, and we get to see exactly which one.

This approach builds on the author's earlier Polyseer project (deep research for Polymarket betting). The upgrade here: instead of one AI against the market, three different "brains" receive the same evidence side by side, making the differences in their reasoning styles clearly visible.

Prediction markets are mechanisms that aggregate collective forecasts through contract trading. Participants buy and sell binary contracts (e.g., "Yes/No") on future event outcomes, with prices fluctuating between 0 and 1 to reflect the market's collective probability judgment. Polymarket runs on blockchain (Polygon network), while Kalshi is a regulated exchange overseen by the U.S. CFTC. Unlike polls or expert forecasts, prediction market participants stake real money, so price signals carry "skin in the game" incentives — incorrect predictions carry real financial consequences, theoretically filtering out empty opinions and better reflecting participants' true beliefs. Academic research (including work by Arrow et al.) has repeatedly demonstrated that prediction markets often outperform traditional methods in forecasting elections, economic data, and similar domains.

"AI vs. the Market" Is Actually a False Premise

The most counterintuitive finding from the experiment: once these models see market prices in their context, they immediately copy the answer.

The author tested this with a question about Fed rate cuts, and the results are telling:

Context	P(Fed cuts rates)
No market price	0.63
"Market pricing 30%"	0.30
"Market pricing 71%"	0.71
"Market pricing 95%"	0.88

In other words, the moment a model sees the odds, its "prediction" becomes little more than a paraphrase of the market. Any "AI vs. market" comparison that doesn't hide the prices first is simply comparing the market against itself.

To make the comparison meaningful, the entire data pipeline strips out all sources referencing odds, betting sites, or FedWatch before the models read anything. This is the core design choice that makes the experiment credible.

This phenomenon is known in machine learning as context anchoring or anchoring bias. When a model's training data contains many instances of "make a judgment after referencing a known number," the model learns a shortcut: treating numbers in context as strong priors rather than reasoning independently. Prediction market odds are particularly high-confidence signals for models — because training data discussing market prices is typically accompanied by narratives like "markets are efficient." This explains why, even when explicitly asked to "judge independently," a model's output will gravitate strongly toward any odds figure present in the context. This closely mirrors anchoring effects in human cognitive experiments, where estimates are significantly biased toward an initial number regardless of its relevance to the question.

The Three Models Have Strikingly Different "Personalities"

With market prices hidden, the three models display sharply distinct reasoning styles. The author provides a replay example using the question "Will the Fed hold rates in October?" (Polymarket was pricing 83% for no change at the time):

Article read	Decisions	Jev	Clef
No news yet	68	51	69
Article 3: Schwab "Fed votes 12-0 to hike"	30	43	8
Article 5: An SEO spam page	18	38	63
Article 8 (final)	17	30	19

The differences are clear:

• Decisions is the most decisive and most willing to diverge from the market;
• Jev barely budges on any piece of information — extremely conservative;
• Clef reacts most dramatically, even getting misled by an SEO spam page (jumping from 8% to 63%), though it self-corrected when the next credible article arrived.

This article-by-article replay mechanism — watching each model change its mind in real time — is arguably the most valuable feature of the project. You can pinpoint exactly which headline moved which model.

Technical Implementation

The system's workflow is cleanly designed:

• Market data: Pulls highest-volume open questions from the Polymarket public API, with search coverage across both Polymarket and Kalshi;
• Evidence collection: Retrieves news from the past 60 days relevant to each question, filters out all content referencing odds or betting markets, and retains the 8 most relevant articles;
• Decision: Sends the same batch of evidence to all three models, each returning P(YES);
• Replay: Re-queries each model with articles 1 through k for each value of k, generating an update timeline per model. Each question produces 27 total decisions, completing the full process in roughly 2 seconds;
• Dashboard: The homepage refreshes via cron every 15 minutes, shared across all users to avoid repeated API calls from idle browser tabs.

The tech stack is built on Next.js 16 + React 19 + Tailwind 4. The OpenAI Decisions API request structure is remarkably clean — just pass in evidence and a yes/no question to get back a probability. Clef is called via Cloudflare Workers AI; since it uses open weights, it can also run locally with Ollama. Jev is accessed through the Vercel AI Gateway. News and evidence retrieval uses the Valyu Search API, which returns full article content for use by the decision models.

Limitations and Open Questions

The author is refreshingly candid about the experiment's limitations: the sample size is tiny, and no markets have settled yet (the Fed example resolves October 28). This is an experiment, not a trading edge — and definitely not investment advice.

The project code is open-sourced under the MIT license, with a hosted version also live. The author also poses an open question worth community discussion: once these markets finally settle, what's the fairest way to evaluate calibration across models?

The project's value may lie less in the accuracy of its predictions and more in what it reveals about a commonly overlooked problem: when evaluating AI decision-making, context contamination — like leaking market prices — can reduce what looks like independent judgment into mere repetition. That's a practical warning for anyone using LLMs for forecasting or decision-making.

Calibration is a core metric for evaluating probabilistic prediction quality — it refers to the consistency between a model's predicted probabilities and the actual frequency of outcomes. A well-calibrated model, when it says it's 70% confident about something, should be correct roughly 70% of the time across many predictions. Common quantification methods include the Brier Score (mean squared difference between predicted probability and actual outcome) and calibration curves (reliability diagrams). The challenge for this project: each prediction market question settles only once, so a single outcome can't distinguish "accurate prediction" from "lucky guess." Making a statistically meaningful calibration comparison across three models requires accumulating data over hundreds of settled questions — which is exactly why the author says the "fairest evaluation method" remains an open question.