Why Did Your Backtest 'Beat' SOTA? The Weather Covariate Leakage Trap

The Illusion of 'Beating' SOTA

Three weeks ago, an engineer applied the Timestral model to day-ahead load forecasting, replacing the previous gradient boosting model. The backtest results were exhilarating: MASE dropped from 0.71 to 0.49. For a moment, he thought he had found a better solution.

But then he discovered that this "victory" came from a hidden error—the weather covariate had leaked during backtesting. The problem wasn't the model itself, but the design of the feature pipeline.

How the Leakage Happened

The feature pipeline joined weather data by timestamp. During backtesting, the pipeline pulled actual temperatures rather than forecast temperatures. More critically, this feature was labeled as "known future," allowing the model to attend to it across the entire forecast horizon. The Timestral paper explicitly states that known future features participate in attention computation across the full horizon.

The problem: in real forecasting scenarios, only weather forecasts are available at prediction time—not actual temperatures. The backtest used future information, artificially inflating performance.

The fix was straightforward: rerun with archived forecast data. The result was MASE 0.66, only a slight improvement over the original 0.71. This is the model's reasonable performance under real conditions.

MASE (Mean Absolute Scaled Error) is a commonly used accuracy metric in time series forecasting, defined as the ratio of forecast error to the error of a naive baseline (such as seasonal naive). A MASE below 1 indicates the model outperforms the baseline, and lower values are better. Its advantage is being scale-independent, facilitating cross-series comparison. However, note that MASE comparability depends heavily on the choice and computation of the baseline—different papers may use different scaling factors, so comparing MASE values from different sources requires caution. In this case, MASE dropping from 0.71 to 0.49 appeared to be a significant improvement, but it rose back to 0.66 after leakage correction, showing that the metric is highly sensitive to feature quality.

Known future covariates are a special class of inputs in time series models—variables whose future values are genuinely known at prediction time, such as calendar features (holidays, day of week) and planned promotional activities. The model can safely leverage these features for attention computation across the entire forecast horizon. However, if actual observations (such as actual temperatures) are mistakenly labeled as known future features, the model will "see" future information, leading to inflated backtest performance. The correct approach is to mark only forecast values or deterministic plans available at prediction time as known future, and to conduct strict temporal availability audits on features.

The 'Time Travel' Trap in Backtesting

This case reveals a common but easily overlooked trap in time series backtesting: feature leakage. When the feature pipeline joins by timestamp, it's easy to inadvertently introduce future information.

According to the engineer, he labeled actual temperatures as "known future" features, which effectively allowed the model to "see" the future at prediction time. In scenarios like load forecasting where weather is a strong covariate, the performance boost from leakage can be very significant.

Similar issues are not uncommon in the industry. Many published SOTA time series results do not detail their data splitting methods, and performance metrics tend to cluster in a narrow range. The engineer noted that in the HyperAI SOTA table, no page explains how the data was split. This increases the difficulty of reproduction and comparison.

GIFT-Eval is a public evaluation benchmark for time series foundation models, designed to provide standardized data splits and evaluation protocols to facilitate fair comparison between models. It covers multiple domains and forecast horizons, and emphasizes zero-shot or few-shot evaluation. In this case, Timestral's MASE on GIFT-Eval was 0.658, close to the engineer's corrected 0.66, while the original leaked result of 0.49 deviated significantly. This suggests a lesson for practitioners: when your own backtest results significantly outperform public benchmarks, you should first suspect data leakage or inconsistent splitting—rather than rushing to declare victory over SOTA.

The Dilemma of Choosing Archived Forecasts

After rerunning with archived forecasts, MASE dropped to 0.66. But the engineer raised another issue: forecast archives have multiple versions per day, and choosing the latest version may still constitute "cheating." The latest version may contain more observational information that was not available at the backtest time point.

Additionally, the archive only covers 14 months, making earlier backtests unavailable. He is rebuilding the backtesting framework to address these issues.

A Checklist for Practitioners

Based on this case, several practical recommendations can be distilled:

1. Verify temporal availability of features: For each feature, clarify whether it is truly available at prediction time. Actual observations are typically unavailable at prediction time—only forecast values are available.
2. Distinguish 'known future' from 'unknown future': If the model supports known future features, you must confirm that the feature is genuinely known in real scenarios. Otherwise, it should be marked as unknown or excluded entirely.
3. Use archived forecast data: When using forecasts as features, use the forecast version available at prediction time, not the latest post-hoc version.
4. Check the splitting details of public results: When citing SOTA numbers, pay attention to whether their data splitting method matches your scenario.
5. Keep backtest logs: Record the feature versions and forecast versions for each backtest to facilitate tracing and reproduction.

Balancing Model Selection and Engineering Reality

Timestral still outperformed the original model in the corrected backtest: 0.66 vs. 0.71. But the improvement was far smaller than the initially reported 0.49. This reminds us that when evaluating new models, we need to distinguish between "backtest performance" and "production performance."

The engineer's experience also reflects a common challenge in the time series field: the gap between public benchmarks and real business scenarios. Timestral's MASE on GIFT-Eval was 0.658, and his corrected result on his own data was 0.66—the two are close. The original 0.49 result deviated significantly, serving as a warning signal of leakage.

For teams preparing to adopt new models, it's recommended to first validate in a backtesting framework that strictly isolates future information before considering deployment. After all, a model that "beats" SOTA in backtesting may have simply peeked at the answers.

One-line Summary

Weather covariate leakage in backtesting inflated MASE; performance dropped after correction, warning that time series evaluation must strictly isolate future information.

Paragraph Summary

The article describes an engineer who replaced a gradient boosting model with Timestral for day-ahead load forecasting, with backtest MASE dropping from 0.71 to 0.49. He then discovered weather covariate leakage: the pipeline used actual temperatures instead of forecast temperatures and labeled them as "known future" features, allowing the model to attend to future information across the horizon. After correction, MASE was 0.66—a limited improvement. The article analyzes the causes of leakage, notes that archived forecast version selection still has bias, and that public SOTA results lack data splitting details. Finally, it provides a checklist: verify temporal availability of features, distinguish known from unknown future, use archived forecasts, check splitting details, and keep backtest logs—emphasizing the distinction between backtest and production performance.