More Imitation Learning Data Makes Things Worse? A Deep Dive into Compounding Errors and Data Quality Traps

Introduction: A Real-World Imitation Learning Dilemma

Recently, a developer shared their experience training an AI for automatic navigation in Genshin Impact on Reddit, sparking considerable discussion about the practical challenges of Imitation Learning. Their goal was straightforward: teach a model to autonomously navigate specific route segments on the map. However, despite recording over 300 training samples, the model's performance remained disappointing — on a roughly 50-second route segment, the success rate was only 30%–40%, with the model veering off course in all other cases.

Even more puzzling was the discovery that adding more training data actually made the model worse. This seemingly counterintuitive phenomenon reveals some of the most typical pitfalls of imitation learning in real-world applications. This article will use this case study to deeply analyze the root causes of imitation learning failures and provide actionable improvement suggestions.

Fundamentals of Imitation Learning

Before diving into the case study, it's important to understand the basic framework of imitation learning. Imitation learning is a branch of machine learning whose core idea is to let an agent learn a policy by observing expert behavior, rather than through trial and error (as in reinforcement learning) or hand-crafted rules. The most basic form is called Behavioral Cloning, which is essentially a supervised learning problem: using the expert's observed states as inputs and the expert's actions as labels to train a mapping function from states to actions.

This approach was first widely applied in autonomous driving. NVIDIA's 2016 end-to-end autonomous driving paper is a classic example — training a model capable of driving on real roads using only paired data of camera images and steering wheel angles. The appeal of imitation learning lies in not needing to design complex reward functions or requiring the agent to perform extensive exploratory interactions with the environment. But behind this seemingly simple approach lie deep theoretical challenges — as this article's case study reveals.

Case Review: What Methods Were Tried?

The developer's exploration process was quite thorough and worth outlining:

• Single-frame training: The model was trained on individual frames rather than frame sequences. They found that using frame sequences actually degraded performance.
• Incremental data scaling: Initially trained with 60 recorded sessions, then added 43 more, but the new data made the model worse. They hypothesized this was because the 43 new sessions had a higher average mouse turning speed, causing inconsistent data distribution.
• Correction sessions: Attempted to record corrective actions, but saw no clear improvement — they suspected the recording or usage method was wrong.
• Route separation: Separately recorded successful samples traveling along the left and right sides of the road, which also made things worse.

These attempts covered multiple dimensions including data volume, data diversity, and error correction mechanisms, but none broke through the bottleneck. Where exactly was the problem?

Compounding Errors: The Classic Failure Mode of Imitation Learning

The most classic failure mode of imitation learning is the Compounding Errors problem, academically known as "distribution shift."

The Theoretical Root of Distribution Shift

Distribution shift is a core concept in statistical learning theory, referring to the mismatch between the training data distribution and the test/deployment data distribution. In the context of imitation learning, this problem is particularly severe: during training, the states the model sees all come from expert trajectories (i.e., the state distribution induced by the expert's policy), but during deployment, the model's own policy induces a different state distribution.

Stéphane Ross et al. provided rigorous theoretical analysis in their classic 2011 paper: for a sequential decision problem of length T, behavioral cloning's error accumulates at a rate of O(T²), not linearly. This means even if the single-step error is very small (say 1%), after 50 steps the accumulated error can become unacceptable. This quadratic error growth rate is precisely the mathematical root cause of why the developer's model started losing control after about 30 seconds.

Why a Single-Frame Model Drifts Off Course

Imitation learning essentially teaches the model a mapping from "expert state → expert action." But during training, the model only sees states generated by the expert that lie on the correct trajectory. Once the model makes a small mistake during actual execution — say, drifting slightly off course — it enters a state that never appeared in the training data.

In this unfamiliar state, the model's predictions become even more unreliable, producing larger deviations, which lead to even more unfamiliar states... errors compound step by step until the model completely leaves the intended path. This is exactly the "starts drifting after 30 seconds" behavior the developer described.

DAgger Algorithm: The Correct Way to Do Error Correction

The developer tried "correction sessions," which was the right direction, but likely implemented incorrectly. The industry standard solution is the DAgger (Dataset Aggregation) algorithm, whose core approach is:

1. Let the current model actually run the game;
2. During the model's execution, have the expert (human) label the "correct action" for the states the model visits;
3. Add these "model's error states + expert corrective actions" to the training set;
4. Retrain and iterate.

The key distinction is: DAgger collects correction data for error states that the model itself actually encounters, rather than having a human intentionally steer off course and then correct back. The latter collects data from a distribution that doesn't match the model's actual error distribution, which is why it often doesn't work.

DAgger was proposed by Stéphane Ross, Geoffrey Gordon, and Drew Bagnell at the 2011 AISTATS conference and is a milestone work for addressing distribution shift in imitation learning. Its theoretical guarantee reduces the error from O(T²) to O(T), i.e., linear growth. DAgger has several practical variants: SafeDAgger allows data collection under safety constraints; HG-DAgger (Human-Gated DAgger) lets humans decide when to intervene; EnsembleDAgger uses model ensemble uncertainty to decide when to request expert labels.

In game AI scenarios, a practical approximation is: run the current model, pause the game at fixed intervals, have a human label the correct action for the current frame, then merge these new data points with the original data and retrain. This is much more efficient than re-recording from scratch because it precisely targets the areas where the model actually makes mistakes.

Data Quality Over Quantity: The Iron Law of Imitation Learning

The phenomenon of "adding 43 sessions made things worse" points directly to another iron law of imitation learning: data consistency and quality matter far more than quantity.

Inconsistent Action Distributions Are Fatal

The developer astutely noticed that the 43 new sessions had faster mouse turning speeds. This means that for the same visual input, different batches of data provided different action labels. Facing these contradictory supervision signals, the model can only learn an "averaged," blurry policy, naturally resulting in degraded performance.

Similarly, "separately recording left-side and right-side routes" may seem to add diversity, but actually introduces a multimodal action distribution: from the same viewpoint in the center of the road, one dataset teaches the model to go left while another teaches it to go right. A standard regression model will predict the average of both — heading straight into obstacles in the center of the road, or oscillating indecisively.

Deep Analysis and Solutions for the Multimodal Problem

The multimodal distribution problem is one of the core challenges in imitation learning. When facing the same or similar observation states, the expert might take multiple different but equally valid actions (e.g., on a wide road, driving on either the left or right side is fine). Standard mean squared error regression will learn the mean of these actions, producing an action that no expert would actually execute.

Solutions include:

1. Mixture Density Networks (MDN): Output a multi-peaked probability distribution of actions rather than a single value, allowing the model to explicitly represent "there are two reasonable choices here";
2. Conditional Variational Autoencoders (CVAE): Model action diversity through latent variables;
3. Diffusion Policy: A recently emerging method that generates actions through a denoising diffusion process, naturally capable of modeling multimodal distributions;
4. The simplest engineering solution — ensure that for the same scenario, only one consistent operating style is kept in the training data, eliminating multimodality at the source.

Practical Tips for Maintaining Consistent Operating Style

• Fix operating habits during recording: Keep turning speed, route following, and camera control methods as consistent as possible.
• Remove low-quality samples: It's better to have 60 highly consistent, high-quality recordings than 300 with mixed styles.
• Normalize actions when necessary: Standardize mouse movement values to reduce interference from absolute speed differences.

State Representation and Sequence Modeling: Why Frame Sequences Made Things Worse

The developer mentioned that "using frame sequences actually made things worse," which is a point worth exploring further.

Limitations of Single-Frame Input

Single-frame input lacks temporal information — the model cannot determine from a single static image whether it's currently accelerating, decelerating, or turning at a constant rate. This is an important gap in navigation tasks. In theory, introducing frame sequences (or temporal structures like LSTM, Transformer) should help.

But why did it get worse in practice? There are two common reasons:

1. Causal Confusion: When the model can see historical frames, it may learn a shortcut — directly predicting the current action based on "the inertia of the previous frame's action" while ignoring the actual navigation cues in the image. Once this "inertia" fails during testing, the model immediately collapses.
2. Insufficient data to support more complex models: Sequence models have more parameters and require more and cleaner data to train properly. At a scale of 60–300 sessions, higher-capacity models are more prone to overfitting to irrelevant features.

The Deep Mechanism of Causal Confusion

Causal confusion was formally proposed and systematically studied by Pim de Haan et al. at the 2019 ICML conference. Its essence is: when the model's observations contain features that have spurious correlations with expert actions, the model may learn to rely on these shortcut features while ignoring the true causal features.

In classic autonomous driving experiments, researchers found that when the model could observe brake light status, it learned the shortcut "decelerate when seeing brake lights ahead" rather than understanding the actual traffic situation ahead. With frame sequence inputs, historical action information (implicit in frame changes) is a strong shortcut signal — the model may learn to simply continue the previous action trend rather than making navigation decisions based on visual cues.

Solutions include information bottlenecks, causal inference frameworks, or in practice, data augmentation techniques such as randomly masking historical frames and applying random temporal offsets to inputs to break these spurious correlations.

Improvement Roadmap: From Short-Term Fixes to Long-Term Optimization

Taking a comprehensive view, here's a viable roadmap for practitioners facing similar imitation learning challenges:

Short-Term Priorities

• Implement DAgger-style online correction: This is the most direct and effective approach to solving the drift problem.
• Unify data quality: Re-examine all recorded samples and remove or correct portions with inconsistent operating styles.
• Return to task fundamentals: First achieve 90%+ success rate on a single route with a single style before considering generalization.

Medium to Long-Term Directions

• Consider reinforcement learning fine-tuning: Fine-tuning with reward signals (such as "distance traveled along the route") on top of the initial policy provided by imitation learning can significantly improve robustness. This "imitation learning + RL fine-tuning" paradigm has become mainstream in the industry — manifesting as RLHF (Reinforcement Learning from Human Feedback) in the large language model domain, and as the pre-train with demonstration data then fine-tune with RL workflow in robotics. Imitation learning provides a reasonable initial policy so that RL exploration doesn't need to start from scratch (avoiding the extremely low exploration efficiency of pure RL); while RL's reward signal can correct the distribution shift and multimodal problems in imitation learning. In game AI scenarios, the reward function can be designed as a combination of distance traveled along the route, penalty for deviation from the route center, and reward for reaching target points. Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) are currently the most commonly used fine-tuning algorithms.
• Improve state representation: If sticking with sequences, be sure to combine them with data augmentation and randomization to avoid causal confusion.

Recommended Learning Resources

For readers who want to systematically understand this field, start searching with these keywords: Behavioral Cloning, DAgger, Distribution Shift, Causal Confusion in Imitation Learning. The imitation learning sections of UC Berkeley's CS285 (Deep Reinforcement Learning) course are excellent introductory material. Additionally, CMU's robot learning courses and Stanford CS 237B (Principles of Robot Learning) provide complete coverage from theory to practice.

Conclusion

This game AI navigation project is a textbook-level "collection of pitfalls" for imitation learning: from compounding errors and distribution shift, to data quality issues and causal confusion, nearly every classic challenge was genuinely triggered. The biggest takeaway is that imitation learning has a deceptively low barrier to entry (just record a few demonstrations), but to truly make it work, understanding these underlying mechanisms is far more important than simply piling on more data.

For any developer looking to apply imitation learning in game AI, robotics control, or similar domains, we hope this case study helps you avoid some detours.