How Overlapping Windows Manufacture Statistical Significance Out of Thin Air: The Invisible Trap in Backtesting

A walk-forward backtest produced what looked like a publishable result: 913 causal anchors, 30-minute directional accuracy. This article explains why it's fake — and a failure mode more dangerous than "the result goes to zero."

The Result That Looked Publishable

Start with the baseline. Many people doing directional forecasting on time series default to 50% as the pass mark. But the baseline you actually need to beat is the majority class proportion. If 51% of moves are up, then "always guess up" gets you 51%. Using 50% as the baseline means mistaking a slight directional bias in the market for model signal.

Here's the result: model 55.2%, majority class 51.1%, net edge +4.1 percentage points, n=912, about 2.4 standard errors. By most review standards, that already crosses the significance threshold.

The problem lies in how the anchors were sampled.

Why It Doesn't Hold Up

The spacing between anchors is about 7 minutes, while the forecast spans 30 minutes.

This means a prediction starting at 10:00 and covering through 10:30, and a prediction starting at 10:07 and covering through 10:37, share 23 minutes of outcome — the vast majority of the entire 30-minute window. They aren't independent observations; they're the same stretch of market action counted over and over. n=912 actually corresponds to far fewer than 912 independent market events.

Standard errors scale with the square root of n. Artificially inflate n, and the error bars get artificially squeezed. This isn't a large amount of data — it's a data count that's been double-counted.

In most financial time-series forecasting tasks, how exactly does sample size get inflated? A frequently overlooked key lies in the underlying implementation of sliding-window evaluation. Take a 30-minute forecast horizon with a 7-minute anchor interval: each forward step of the anchor generates one sample containing features and labels, but these samples' label windows overlap heavily, causing the same stretch of price action to be repeatedly treated as independent evidence. In machine learning practice, data leakage and sample duplication belong to the same class of systematic bias — both blur the information boundary between validation and training sets, or cause excessive information sharing within the validation set. Many backtesting frameworks generate anchors at a fixed step by default and never check for actual independence, which means researchers can inadvertently manufacture thousands of highly correlated observations even when using strict walk-forward splits, compressing the standard error estimate below its true level.

Rescoring on Non-Overlapping Anchors

Swap the same evaluation to non-overlapping anchors and look only at independent observations:

Scheme	n	Majority Class	Model	Net Edge	Standard Errors
Overlapping	912	51.1%	55.2%	+4.1 pp	2.4 s.e.
Non-overlapping	163	54.0%	59.5%	+5.5 pp	1.4 s.e.

Notice what happened: the edge actually got bigger, rising from +4.1 to +5.5 percentage points, but significance nearly halved, dropping from 2.4 standard errors to 1.4. The inflation factor is about 2.4x, exactly sqrt(912/163).

This is precisely what makes the trap so insidious.

The Trap's Real Lethality: The Effect Size Grows

If rescoring collapsed the edge to zero, anyone would immediately declare the result dead. But in reality it improved, which tempts you to conclude "the true effect is actually stronger, I just need more data."

A larger estimate on a smaller sample is exactly what noise looks like. The smaller the sample and the larger the variance, the more chances for extreme values to appear. You think you've captured signal, when really you just drew a longer tail.

The original author's summary is blunt: this isn't a problem of statistical power, it's a problem of effective sample size being overestimated.

Why do estimates get bigger on smaller samples? This involves the variance-precision tradeoff in statistical estimation. With small sample sizes, the sampling distribution is wider, extreme values are more likely, and the estimator tends to deviate from the true population value. When observations are mutually independent and the sample is large enough, the central limit theorem guarantees the estimator converges around the true value, with variance shrinking as n grows. Conversely, overlapping windows nominally increase n, but the effective sample size doesn't grow, so the estimator's fluctuation remains large. Non-overlapping anchors drop the sample size from 912 to 163; although the variance of a single estimate rises, the estimator is no longer repeatedly weighted by the same stretch of market action, and thus comes closer to the true effect. However, because small samples are more prone to larger point estimates, a researcher seeing the +5.5pp edge may misread it as a stronger signal — which is exactly the signature of noise, not evidence of insufficient power.

A Check You Can Run Immediately

To determine whether overlap contamination exists, you only need to compare two numbers: anchor interval vs. forecast horizon.

• Interval smaller than horizon → overlap exists, n is inflated
• Interval equal to or greater than horizon → observations are mutually independent

If you've already run the results, you can rescore afterward with non-overlapping anchors. A safer approach is to lay out anchors at intervals of one full horizon from the start, eliminating overlap at the source.

This check isn't limited to financial backtesting. It applies to any sliding-window evaluation where "the window is wider than the step": sensor data processing, power load forecasting, anomaly detection, medical monitoring — as long as the evaluation uses a sliding window wider than the stride, the same inflation occurs.

A Bit of Extended Thinking

The overlapping-window problem is methodologically a form of pseudoreplication. Biostatistics and ecology have long discussed it systematically, but in engineering and quant circles it's often overlooked because sliding-window evaluation is so natural to write — rolling forecasts, point-by-point scoring, a few lines of code and it runs, and nobody checks whether the anchors are mutually independent.

A useful self-check habit: whenever you see any time-series result with "a big n and high significance," first ask — among these n observations, how many share the same outcome window? The answer will often make the error bars grow back.

Source: Michael Hairetis's experiment log, original link.

One-line Summary

Overlapping windows inflate the sample size, causing backtest significance to be inflated and the effect size to be even more misleading.

Paragraph Summary

Through a walk-forward backtest case, the article reveals how overlapping anchors manufacture false statistical significance. In the benchmark test, the model's directional accuracy was 55.2% versus a majority class of 51.1%, a net edge of +4.1 percentage points, with n=912 appearing significant. But because the anchor interval is about 7 minutes while the forecast horizon is 30 minutes, adjacent anchors share most of their outcome window — the observations aren't independent, and the effective sample size is severely overestimated. Switching to non-overlapping anchors drops n to 163, and the net edge actually rises to +5.5 percentage points, but significance plunges from 2.4 standard errors to 1.4. This pattern of growing effect size with falling significance is highly deceptive, easily leading people to think they need more data rather than to fix their evaluation method. The author offers a check: compare the anchor interval to the forecast horizon; if the interval is smaller than the horizon, overlap contamination exists. The trap applies to any sliding-window evaluation where the window is wider than the step.