Why More Accurate Traffic Predictions Fail to Improve Signal Control: A Layered Diagnostic Study

The Paradox: Better Predictions, No Better Decisions

Intuitively, if we can predict traffic flow more precisely, signal control strategies should improve accordingly. But a recent study published on arXiv (arXiv:2610.06992v1) sharply challenges this assumption: better traffic predictions do not necessarily lead to better signal control decisions.

Using real-world traffic data from Xuancheng, China — 29 days of reconstructed demand data with 7 days held out for testing — the research team built a layered diagnostic framework to systematically trace how "predictive value" dissipates at each stage of the forecast-to-decision pipeline. The contribution here isn't yet another stronger prediction model; it answers a commonly overlooked engineering question: at which step does improved prediction fail to translate into real-world gains?

Predictions Did Improve — But Only Marginally

The study first validates the prediction models themselves. Entry-level and movement-level predictions reduced mean absolute error (MAE) by 4.03% and 3.92%, respectively, compared to a historical mean baseline — confirming that the models learned meaningful patterns with measurable accuracy gains.

For uncertainty quantification, the researchers used conformal prediction intervals. A nominally 90% confidence interval achieved 90.72% marginal coverage overall, performing as expected. But a problem immediately emerged: on the retrospectively identified high-demand subset, coverage dropped sharply to 75.66%. In other words, precisely during the periods of highest traffic stress — when accurate decisions matter most — the prediction intervals became far less reliable. This is a classic case of "performs well on average, fails at critical moments."

Conformal Prediction is an uncertainty quantification framework that provides statistical coverage guarantees without strong distributional assumptions. Its core idea is to attach a confidence interval to predictions that is guaranteed to cover the true value with a specified probability (e.g., 90%) over the long run. Unlike Bayesian intervals, conformal prediction's coverage guarantee is a frequentist marginal guarantee — valid over the entire test set, but not necessarily over any specific subset (such as high-demand periods). This is exactly the problem observed here: the global 90.72% coverage masks the 75.66% coverage failure on the high-demand subset. This lack of conditional coverage is often more dangerous in deployment than average error, because the system loses its statistical guarantee precisely in the extreme scenarios where reliable predictions are needed most.

Interface Audit Reveals Two Hidden Structural Flaws

The most illuminating part of the study is an audit of the "interface" between prediction and control. The team identified two structural issues capable of nullifying all predictive value.

Decision-Time Information Leakage

The first issue is decision-time leakage. In the original experimental setup, the controller effectively "peeked" at future information that should not have been available at the moment of decision-making. This temporal misalignment systematically inflates the estimated gains from prediction — because what's actually being evaluated is a cheating version of the controller. The study corrected this temporal interface to ensure fair evaluation.

A Scarcity of Actionable Choices

The second issue is more fundamental: across 9 controlled intersections, only 2 actually offered multiple distinct, effective actions. This means that at most intersections, no matter how accurate the predictions are, the controller has very few choices to make. No matter how rich the predictive information, if there aren't enough distinguishable decision actions to absorb it, it has nowhere to go. This reveals that action identifiability is the hard bottleneck constraining predictive value.

The concept of an "interface audit" comes from integration testing in software engineering. In prediction-driven control systems, it refers to a systematic inspection of the data exchange protocol between the prediction module's output and the control module's input. Prediction models are typically trained and evaluated offline in isolation, while controllers make real-time decisions online — creating multiple potential misalignment points around timestamp alignment, data granularity matching, and information availability boundaries. Without rigorous interface auditing, researchers can easily misattribute the results of a joint "prediction + control" evaluation to prediction quality alone, when gains or losses may stem entirely from misconfigured interfaces. This is one common reason why prediction models that excel in the lab see dramatically reduced benefits in real-world deployment.

Synthetic Positive Control: Future Information Can Be Highly Useful

To prove the methodology itself is sound, the study designed a synthetic positive control. In this controlled environment, future information reduced internal rollout costs by 61.5%. This confirms that when conditions are ideal, foreknowledge of the future does deliver enormous benefits — the problem isn't that future information is useless, but that real-world system constraints prevent it from being realized.

A Synthetic Positive Control is an experimental design strategy borrowed from clinical trial methodology, where a positive control group validates that the measurement framework can detect a known true effect. The logic is: before formally testing a method, first verify that the entire evaluation framework can detect real effects under artificially constructed ideal conditions. If even ideal conditions yield no detectable benefit, the evaluation framework itself is flawed. Only when ideal conditions confirm effectiveness can failures under real conditions be attributed to practical constraints. In this study, the 61.5% cost reduction in the synthetic control confirms that the simulation environment and optimization algorithm are functioning correctly, allowing subsequent real-world failures to be attributed specifically to structural system constraints (scarce actions, interface misalignment, etc.) rather than methodological flaws. This "first prove it can work, then diagnose why it doesn't" design philosophy is worth adopting more broadly in AI system evaluation research.

Real-World Testing: Causal Predictions and the "Event Oracle" Both Fail

On frozen test dates, the study compared the causal prediction model against a "five-second event oracle" — essentially an idealized near-perfect short-term predictor. The results are striking:

• Compared to a matched baseline with no future information, both the causal predictor and the event oracle actually increased queue vehicle-seconds by 6.09% and 3.39%, respectively;
• The event oracle did reduce spillback exposure by 3.78%;
• But paired-day bootstrap confidence intervals all crossed zero, meaning these differences are not statistically significant.

In other words, even when the controller is given near-perfect short-term predictions, it cannot reliably improve core operational metrics under real-world constraints. This stands in sharp contrast to the large positive gains in the synthetic control, powerfully demonstrating that the problem lies in structural constraints within the real system — not in prediction capability itself.

Predictive Value Depends on Four Prerequisite Conditions

Synthesizing all experiments, the study distills four key dimensions that determine whether predictive value can be realized in practice:

1. Temporal observability: Can the required information be legitimately accessed at the time of decision-making?
2. Action identifiability: Does the controller have enough distinct, effective actions to choose from?
3. Dynamics consistency: Do the dynamics of the simulation or control environment match reality?
4. Objective alignment: Does the optimization objective align with the operational metrics that truly matter?

If any single link in this chain breaks, improvements in prediction accuracy may fail to translate into operational gains — or may even backfire.

Implications for Intelligent Transportation Research

The significance of this research extends well beyond traffic signal control. It provides an actionable diagnostic protocol for the broad class of forecast-to-decision problems, helping researchers and engineers pinpoint exactly where prediction improvements are being lost.

For AI deployment scenarios across the board — supply chain scheduling, energy management, or any other prediction-driven decision system — this finding carries a universal warning: chasing lower prediction error is not the same as chasing better decision outcomes. Before investing heavily in refining prediction models, it is often far more important to first examine whether the downstream decision system is actually capable of absorbing predictive value. Model accuracy is only one link in the chain, and real-world gains are determined by the weakest link.

The Forecast-to-Decision (F2D) problem has a broader theoretical background in operations research and reinforcement learning. Traditional machine learning optimizes for prediction error (e.g., MAE, RMSE), while Decision-Focused Learning (also known as Predict-then-Optimize) advocates using downstream decision quality directly as a training signal, closing the gap between prediction accuracy and decision benefit. Yet even with decision-focused learning, the structural issues this paper identifies — insufficient action identifiability and interface misalignment — still block value transmission. This shows that the problem exists not only in the choice of loss function, but at the system architecture level. Before deploying any prediction-based AI decision system, systematically verifying that the downstream execution layer has the capacity to "absorb" predictive information is a step that is routinely skipped in engineering practice — yet it is absolutely critical.