138k Financial Headlines Fine-Tune ModernBERT: Training Loss Drops, Test AUC Stuck at 0.50

A Reddit user fine-tuned ModernBERT on 138,000 financial news headlines to predict which recently fallen stocks would rebound more strongly. The result was blunt: training loss kept dropping, but test-set AUC stayed stuck between 0.49 and 0.51 — essentially a coin flip.

The value of this "failed" experiment may be greater than a nice high score would have been. It exposes several of the most easily overlooked pitfalls in financial text prediction — target leakage, near-duplicate samples, and non-stationarity — with striking clarity.

Experimental Design: Stripping Market Factors Out of the Label

The dataset comes from Benzinga and contains 137,934 news headlines covering 2,518 US stocks. Each sample uses that stock's headlines from the previous 14 days as input. After filtering, there are 115,154 cases, all of them stocks with "good earnings but recently falling prices" — superficially similar conditions, and the real question is: which ones will recover better?

The target isn't a simple up-or-down movement, but a simulated trade: it ends when the price move reaches 3× the stock's daily volatility, or after 20 trading days. Results are measured in units of "daily volatility multiples," avoiding high-volatility stocks dominating the metric.

The key step is subtracting the average return of all cases on the same trading day. The reason is straightforward: how the whole market moves next week isn't contained in any single news headline. Without this subtraction, the model would likely just be learning market sentiment rather than the relationship between headlines and individual stock performance.

The label is finally defined as: whether the case outperformed the daily median. So AUC measures: among samples from the same day, can the model separate the better half from the worse half?

ModernBERT is an encoder architecture released in late 2024, essentially a modernized upgrade of BERT. It uses rotary position embeddings (RoPE) instead of absolute position embeddings, uses FlashAttention for acceleration, and extends the maximum context window from BERT's 512 tokens to 8192 tokens, while improving inference efficiency through tricks like unpadding and weight sharing. For short-text classification tasks like financial news headlines, ModernBERT's advantages lie in more stable long-text modeling and faster training, which is why it's widely adopted in fine-tuning experiments. However, architectural improvements can't solve label noise and non-stationarity — which is precisely the core finding of this experiment.

AUC (Area Under the ROC Curve) is a common metric for measuring a model's ranking ability in binary classification tasks, with 0.5 meaning no better than random guessing and 1.0 meaning perfect separation. In this article, AUC measures: among samples from the same day, can the model separate the better-performing half from the worse-performing half? The reason it's computed within same-day samples is that the market average return has already been subtracted, making cross-day comparisons meaningless. AUC stuck at 0.49–0.51 means the model's ranking ability is not statistically significantly different from a coin flip — even though training loss is dropping, the model is just fitting noise rather than learning a generalizable signal.

Preventing Leakage: Strict Time-Based Splitting

The author did several things that are mandatory in financial machine learning:

• Strict time-based splitting: training set cut off at November 2024, test set starting from January 2025, with a gap in between to ensure no label overlap.
• Only one case per stock per week. Adjacent trading days share nearly identical headlines and nearly identical outcomes; if every day were kept, the model would memorize stocks rather than learn patterns.
• This kind of "time-based deduplication" is not optional — it's the baseline.

Even with all these measures, the results still didn't beat random.

Near-duplicate sample leakage is a hidden but deadly pitfall in financial text modeling. News headlines on adjacent trading days are often highly similar or even identical (e.g., "Company announces buyback plan" may appear for several consecutive days), and the corresponding return labels are also highly correlated due to overlapping holding periods. If you randomly split training and test sets, the model will have "seen" close relatives of test samples during training, leading to inflated test performance. More insidiously, even with time-based splitting, the same stock appearing repeatedly in both training and test sets means the model may memorize stock-specific patterns rather than learning the general relationship between news and returns. Therefore, time-based deduplication (such as keeping only one case per stock per week in this article) is a necessary step to prevent leakage, not an optional optimization.

Results: Fine-Tuning, Keywords, and kNN All Fail

The author compared multiple approaches:

• Fine-tuned ModernBERT (5-day and 14-day windows, with or without industry information in the text): AUC between 0.49 and 0.51. Training loss kept dropping; test performance was always coin-flip level.
• Handcrafted keyword features (upgrades, guidance cuts, buybacks, etc.): almost all effects flipped sign between training and test sets. For example, "analyst upgrade" was +0.28 before 2025, then became -0.04 after 2025.
• kNN based on bge-base embeddings (predicting using the outcomes of the 50 most similar historical cases): all quintiles performed flat with no discrimination.
• Putting text and price features together into a gradient boosting model: no better than random stock selection.

The only pattern that held up in both periods was a narrow one requiring manual cleaning: companies performed significantly worse after issuing new shares. But there were only 98 such samples — too small to serve as a reliable strategy.

Three Lessons Worth Remembering

Dropping Training Loss Means Nothing on Noisy Targets

Labels in financial prediction are inherently extremely noisy. A model can happily fit noise, with training loss steadily declining, but test AUC won't budge. Seeing loss drop and assuming the model has learned something is a dangerous form of self-comfort.

Leakage from Near-Duplicate Samples Is Easy to Overlook

Adjacent trading days share many headlines, and outcomes are also highly correlated. Without deduplication, the model is effectively "memorizing stocks" rather than learning the relationship between news and returns. Time-based deduplication is the most easily skipped — and most un-skippable — step in financial text modeling.

Keyword Effects Can Flip Entirely

The author says the keyword sign flip is the clearest picture of non-stationarity he's ever seen. Patterns that worked in 2023 may directly reverse afterward. Extrapolating patterns found in the training period directly to the test period is especially dangerous with financial text.

Community Suggestions: Ranking Loss and Shorter Horizons

The author posed a question to the community at the end: is this kind of zero result common in financial text prediction? Suggestions included switching to per-date ranking loss, computing daily rank IC, and shortening the prediction horizon to 1–3 days.

These directions are worth trying. Per-date ranking is closer to the real question of "who's better on the same day," and shorter horizons also reduce interference from other information during the holding period. But before trying more models, getting data leakage and time-based splitting right may be more important than switching models.

This experiment didn't produce a tradeable signal, but it produced a very practical failure checklist: targets must be market-neutralized, samples must be time-deduplicated, loss dropping doesn't mean learning patterns, and keyword effects drift. For those working in financial NLP, these are more valuable than an inflated backtest curve.

Per-date ranking loss is a training objective designed for panel data. Traditional binary classification loss treats each sample independently, but the core question in financial prediction is "among stocks on the same day, which are relatively better." Per-date ranking loss doesn't require the model to predict absolute up or down movements, but rather to give a ranking consistent with actual returns for samples within the same day. Common implementations use pairwise ranking loss (like RankNet) or listwise loss (like ListNet), computing loss with each day as a group. The advantage is that it automatically strips out overall market movements and focuses only on cross-sectional discrimination. Combined with daily rank IC (information coefficient), it can more directly measure whether the model has learned useful ranking signals rather than fitting noise in absolute returns.

One-line Summary

138k financial news headlines fine-tune ModernBERT: training loss drops but test AUC is only 0.5, revealing three major pitfalls in financial text prediction.

Paragraph Summary

A Reddit user fine-tuned ModernBERT on 138,000 financial news headlines to predict the rebound strength of recently fallen stocks. Despite steadily declining training loss, test AUC stayed at 0.49–0.51, equivalent to random. The author prevented leakage through subtracting same-day market average returns, strict time-based splitting, and weekly deduplication per stock, but fine-tuned models, keyword features, kNN, and gradient boosting all failed. Handcrafted keyword effects flipped sign between training and test periods, and the only stable pattern — worse performance after new share issuance — had just 98 samples. The experiment's value lies in exposing three common pitfalls: target leakage, near-duplicate samples, and non-stationarity, along with a practical failure checklist.