Offline-Trained Neural Networks Fail in Control Loops? This Paper Offers a Systematic Explanation

An Overlooked Paradox: Learning Changes the World, and Then What Is Learned Gets Changed Too

This sounds like a tongue twister, but it precisely captures a deep-seated contradiction in control systems. Neural networks are introduced into control loops because traditional modeling methods often fall short when dealing with complex nonlinear systems—you can hardly write out the exact dynamic equations for a quadrotor drone in gusty winds, but a neural network can learn an approximate control policy from data.

The problem arises at the moment of deployment. Once a network trained on an offline dataset is connected to a closed-loop control system, every action it takes changes the system's subsequent state distribution—and this new distribution is precisely what it never saw during training. In other words, while the network changes the world, it also changes the very world it depends on.

This paper systematically dissects this paradox and proposes a unified classification framework to explain the failure modes that sequentially emerge during closed-loop deployment.

Closed-Loop Control refers to feeding the system output back to the controller via sensors, which then adjusts the input accordingly, forming a "sense—decide—act—re-sense" cycle. Unlike open-loop systems, the state distribution in a closed-loop system is not fixed but jointly determined by the controller policy and the plant dynamics. This characteristic gives rise to a core concept in control theory—Distribution Shift: once a learned controller (such as a neural network) intervenes in the closed loop, every action it outputs alters the system's future state visitation distribution, causing the actual runtime data distribution to gradually deviate from the offline training data distribution. This phenomenon is ubiquitous in offline reinforcement learning, imitation learning, and neural control, and is the technical root of the paradox that "learning changes the world, and the world in turn changes what is learned."

Three Failure Modes: A Unified Classification Framework

The paper categorizes the failures of offline-trained networks in closed-loop deployment into three progressive stages:

Closed-Loop Identification Error appears in the early deployment phase. During offline training, the network learns the input-output mapping from open-loop data, but in a closed-loop system, feedback coupling exists between the controller and the plant, causing the actual state distribution to deviate from the training set.

Deployment-induced Action Error follows closely. When the network outputs actions based on the shifted state distribution, these actions themselves further push the system away from the training trajectories.

Continuous Tracking Error is the end result. The first two errors accumulate continuously, causing the system to fail to stably track the target trajectory.

The value of this classification framework lies in decomposing problems that are often loosely attributed to "the model isn't good enough" in practice into components that can be diagnosed and intervened upon separately.

The key difference between open-loop and closed-loop data lies in feedback coupling. During open-loop training, the input-output pairs the network receives come from a pre-collected static dataset, and state transitions are not affected by the network's actions; after closed-loop deployment, however, the network's actions change the plant state, which in turn changes the input the network observes at the next time step. Therefore, even if single-step predictions are very accurate on offline data, multi-step errors under closed loop will accumulate and amplify due to feedback. This distinction is known in control theory as the difference between open-loop identification and closed-loop identification; ignoring it leads to systematic bias in the early deployment phase, which is what the paper defines as closed-loop identification error.

Sensitivity Analysis: Why Local Optima Lead to Trajectory Divergence

Through discrete-time sensitivity analysis, the paper provides an important theoretical conclusion: parameters learned offline may be locally optimal near the training data, but once deployed in a closed-loop system, due to the local geometry of the loss function, these parameters will precisely guarantee trajectory divergence.

The intuition is this: the offline training loss function measures prediction accuracy on the training data distribution, but after closed-loop deployment, the actual state distribution the system experiences has shifted. On the shifted distribution, the original "optimal" parameters may lie in the ascending direction of the loss surface rather than the descending direction. The more "confidently" the network acts according to the policy learned during training, the faster the deviation is amplified.

This explains why some models that perform excellently on validation sets quickly become unstable once entering an actual control system—it's not that the model isn't good enough, but that there is a structural misalignment between the optimization objective and the deployment objective.

Discrete-Time Sensitivity Analysis is a tool in control and dynamical systems theory for quantifying how much a system's output responds to small changes in parameters or initial conditions. Its core is computing the partial derivatives of trajectories with respect to parameters (i.e., sensitivities) and recursively propagating these partial derivatives along time steps. In closed-loop systems, sensitivity analysis can reveal whether trajectory errors converge or diverge when learned parameters deviate from the training optimum. The paper uses this tool to prove that certain parameters that are locally optimal under the offline loss function correspond to divergent directions in the closed-loop sensitivity sense. In other words, the two objectives of optimizing the offline loss and guaranteeing closed-loop stability are not consistent; the former may even actively select parameters that lead to trajectory divergence.

Solution: Time-Scale Separated Online Adaptation

The core of the strategy proposed in the paper is "time-scale separated online adaptation." The basic idea is to let the neural network continuously perform bounded learning updates after deployment, rather than freezing parameters. The key lies in "time-scale separation"—the speed of online adaptation must be slow enough not to interfere with the fast dynamic response of the control loop, while being fast enough to track changes in the state distribution.

Ablation experiments on a nonlinear Duffing oscillator validated the effectiveness of this strategy: deployment with bounded continual learning successfully suppressed trajectory divergence, while static deployment failed without exception.

The implication for practitioners is direct: if you plan to put machine learning components into safety-critical feedback systems, completing offline training is not the end—you need to design a constrained online learning mechanism so that the model can continue to adapt to the new world it has created itself after deployment.

Time-Scale Separation originates from singular perturbation theory, referring to the ability to analyze and design controllers separately at different time scales when fast and slow variables coexist in a system. In the online adaptation strategy of this paper, the control loop (such as attitude control and motor response) belongs to the fast time scale and requires high bandwidth and low latency; while the parameter updates of the neural network belong to the slow time scale, achieved through smaller learning rates or bounded update step sizes. As long as parameter updates are slow enough, one can approximately assume that parameters remain unchanged within one response cycle of the fast loop, thereby avoiding mutual interference between online learning and feedback control. This idea has long been applied in adaptive control, but combining it with bounded continual learning of neural networks is the key design in this paper for solving the post-deployment distribution shift problem.

Implications for Engineering Practice

The paper's contribution is not just theoretical analysis, but more importantly providing actionable design principles for engineering practice. For engineers working in robot control, autonomous driving, industrial process control, and other fields, the reminder of this work is: offline training validation metrics (such as test set MSE) may severely overestimate actual performance in closed-loop scenarios. When evaluating a controller, it must be tested in the closed-loop circuit, rather than only looking at open-loop prediction accuracy.

Another practical implication worth noting is: online adaptation is not a panacea; it requires carefully designed constraint boundaries. Unrestricted online learning may introduce new instabilities, and only bounded updates combined with reasonable time-scale design is the key to achieving a balance between adaptability and stability.

One-line Summary

Offline-trained neural networks deployed in closed-loop control diverge due to distribution shift, requiring time-scale separated online adaptation.

Paragraph Summary

The article revolves around the failure paradox of offline-trained neural networks in closed-loop deployment: while the network changes the plant's state distribution, it also changes the training data distribution it depends on. The paper proposes a unified classification framework that categorizes failures into three progressive modes: closed-loop identification error, deployment-induced action error, and continuous tracking error. Through discrete-time sensitivity analysis, the authors prove that offline optimal parameters may precisely cause trajectory divergence in closed loop, rooted in the structural misalignment between optimization objectives and deployment objectives. The solution is time-scale separated online adaptation: allowing the network to continuously perform bounded learning updates after deployment, with update speeds slow enough to avoid interfering with the control loop yet fast enough to track distribution changes. Ablation experiments on the Duffing oscillator validated that this strategy can suppress divergence. The article concludes by pointing out that offline validation metrics severely overestimate performance under closed loop, and safety-critical systems must design constrained online learning mechanisms.