Local Gemma vs. GPT-5.4: Both Lost to a Bot That Just Says 'Up' Every Day

A Four-Month Stock Prediction Experiment

A developer shared on Reddit the results of a four-month experiment: having multiple large language models predict the daily up/down direction of stocks, then rigorously comparing each prediction against real prices. What made this experiment stand out was its transparency — every prediction was locked with a timestamp at submission, nothing was deleted, and all results were recorded on a public leaderboard (LDBD).

The lineup was representative: a 4-bit quantized Gemma 26B running locally on a laptop (built on MLX rather than Ollama, but same idea), plus Claude Opus 4.8 and GPT-5.4 accessed via API. All three received the exact same prompt: given recent closing prices, answer "up" or "down."

The prompt was deliberately kept as bare-bones as possible, designed to test each model's raw predictive ability under the most basic conditions.

The Embarrassing Result: No AI Beat the "Dumb Rule"

The results were a cold splash of water on AI prediction hype.

The locally running Gemma 26B achieved an accuracy of just 49.2% across 453 scored predictions — slightly worse than a coin flip. But the real gut-punch came from the baseline: a dumb rule that answered "up" every single day, applied to the exact same prediction targets, hit 53.4% accuracy.

Before anyone could dismiss the local small model, the author had already closed that escape route — the flagship API models also failed to beat their own baselines:

• Claude Opus 4.8: 53.9%, while its paired "always up" baseline was 54.6%
• GPT-5.4: 50.5%, while its paired baseline was 51.6%
• Local Gemma 26B: 49.2%, paired baseline 53.4%

In other words, not a single model could beat the bot that just says "up." Whether it was a locally run open-source model or an expensive top-tier commercial API, none of them demonstrated any predictive edge over a random baseline when given a naive prompt.

This result isn't surprising — it confirms once again that short-term market direction prediction is essentially a random walk. And the statistical reality that stocks rise slightly more than 50% of the time in a bull market makes "always say up" a surprisingly hard benchmark to beat.

The Only Variable That Worked: The Input Pipeline, Not the Model

The one bright spot in the experiment pointed toward something more worth thinking about.

The author tried changing the input pipeline: instead of feeding Gemma a fixed list of stocks, he picked one stock each day that had news-driven activity. This single change caused the same local Gemma model to outperform its paired baseline by 7.6 percentage points (p≈0.02).

The author himself remained appropriately cautious: the sample size was small, and the significance could be a statistical fluke from multiple comparisons. Even so, the signal was hard to ignore.

It points to a core conclusion:

So far, the input pipeline matters far more than the model itself.

Same model, same parameters — just by changing the input from "static historical prices" to "today's news-driven stocks," performance changed fundamentally. This suggests that a model's predictive power doesn't come from its "intelligence" but from whether the input signal actually contains useful information.

What This Experiment Tells Us About AI Applications

LLMs Are Not All-Knowing Oracles

The most direct value of this experiment is throwing cold water on the popular fantasy of "using LLMs to trade stocks." No matter how large or advanced a model is, it doesn't have magical advantages when asked to predict the direction of inherently random short-term market moves. A top-tier API and a local small model ended up at essentially the same starting line on this task — and neither could beat a rule you could write in one line of code.

The Data Pipeline Sets the Ceiling for Any AI System

The real lesson is this: in practice, how you prepare the input for your model often matters more than which model you choose. This aligns closely with the general experience of AI engineering — retrieval quality, context construction, information filtering — these "unglamorous" engineering steps are frequently what determine whether a system actually works.

A Rigorous AI Evaluation Framework Worth Learning From

The evaluation framework the author built is itself valuable: timestamps locked at submission, results non-deletable, always compared against a paired baseline, statistical significance and sample limitations clearly noted. This "cheat-proof" approach to open evaluation is exactly what's scarce in current AI capability assessments. Too many "AI beats the market" claims lack exactly this kind of comparison baseline and transparent record-keeping.

An Open Invitation: Can Your Local Model Beat the Baseline?

The author openly acknowledges being the builder of the leaderboard (LDBD), and sincerely invites others to participate — he'd love to see someone's local model finally beat the "always up" paired baseline. To help, he's provided a ready-to-run starter bot — about 120 lines of code, only standard library dependencies, dry-run mode by default, open-sourced under the MIT license.

This experiment reminds us, with a touch of dark humor, that embracing the capabilities of large models and maintaining a clear-eyed awareness of their limits are equally important. Sometimes a bot that says "up" every day is the best reality check of all.