Large-Step Adversarial Training: New Findings on Implicit Bias, Optimization, and Sharpness

Adversarial Training (AT) is an important method for improving model robustness. Previous theoretical analyses of gradient descent (GD) dynamics in AT have mostly focused on the continuous-time limit with small or vanishing step sizes, but practical training often uses relatively large constant step sizes. A recent paper specifically studies GD dynamics in AT under large step sizes, revealing three key differences from Standard Training (ST).

Implicit Bias: The Gap Between AT and ST Persists Under Large Step Sizes

The paper first analyzes the convergence point (implicit bias) and loss descent speed (optimization) of GD in adversarial linear logistic regression. Prior work established an implicit bias gap between AT and ST under small step sizes, but whether this conclusion holds under large step sizes was unclear. This study shows that even with arbitrarily large step sizes, the implicit bias gap between AT and ST persists. This means that the large step sizes commonly used in practice do not eliminate the essential difference in convergence behavior between the two training modes.

This finding has direct implications for understanding the source of robustness: the solution obtained by AT systematically deviates from the ST solution in parameter space, and this deviation does not disappear as the step size increases, indicating that implicit bias is an intrinsic property of AT rather than an artifact of the small-step-size limit.

Implicit Bias refers to the phenomenon where optimization algorithms such as gradient descent, while fitting the training data, tend to converge to a particular class of solutions (e.g., minimum-norm solutions). Even when model capacity is sufficient to interpolate the data, the discrete update rule of the optimization algorithm introduces a preference in solution space, determined by the algorithm itself rather than by an explicit regularization term. In standard training, small-step GD tends to converge to the minimum L2-norm solution for linear models; in adversarial training, the inner maximization changes the effective loss landscape, causing GD to converge to a different class of solutions. One of the core contributions of this paper is to prove that this implicit bias difference introduced by AT holds in the small-step-size limit and persists under arbitrarily large constant step sizes, demonstrating that it is an intrinsic property of AT.

Adversarial Linear Logistic Regression is a commonly used simplified model for analyzing adversarial training theory: the model is a linear classifier, the loss is the logistic loss, and the inner adversarial perturbation maximizes the loss within a given norm ball. This model is simple enough to analytically characterize the convergence point and dynamics of GD while preserving the core structure of adversarial training—"inner maximization, outer minimization." This paper rigorously establishes the persistence of the implicit bias gap between AT and ST under large step sizes in this model, providing a theoretical foundation for understanding AT behavior in more complex neural networks.

Optimization Error: No Unified Convergence Rate

In standard training, the optimization error of GD typically has a unified convergence rate guarantee. However, the paper points out that no such unified convergence rate exists in AT. This difference stems from the inner maximization process of adversarial training—the generation of adversarial examples changes with the parameters, making the optimization landscape more complex and the convergence behavior dependent on the specific problem instance.

The implication for practitioners is that one cannot simply apply the convergence intuition from ST to set the training schedule for AT; the convergence speed of large-step AT may vary with data and model, requiring more careful tuning and monitoring.

Convergence Rate describes the speed at which optimization error decreases with the number of iterations, typically given in forms such as O(1/T) or O(1/√T), and is a theoretical metric for measuring algorithm efficiency. In standard training, GD on smooth convex problems has a unified convergence rate guarantee that does not depend on the specific data distribution. However, in adversarial training, the adversarial examples generated by inner maximization change dynamically with parameter updates, causing the effective objective function to be non-convex and evolve across iterations, making it difficult to establish rate bounds that are consistent across all problem instances. This paper points out that no unified convergence rate exists in AT, implying that in practice the convergence speed of large-step AT may be highly dependent on data and model, and one cannot simply apply ST convergence intuition to set the training schedule.

Two-Phase Sharpness Dynamics: Different from Edge-of-Stability

The paper further examines the evolution of loss sharpness during AT in neural network training. In standard training, large-step GD often exhibits "Edge-of-Stability" behavior—sharpness fluctuates around a critical value. In AT, however, sharpness can exhibit two-phase dynamics, different from the edge-of-stability behavior observed in ST.

This two-phase dynamic indicates that adversarial training shapes the loss surface in a fundamentally different way from standard training, and sharpness changes cannot be explained solely within the ST framework. For practitioners concerned with the relationship between generalization and robustness, this suggests the need to design sharpness-related regularization or early stopping strategies specifically for AT.

Edge-of-Stability is a recently discovered phenomenon in large-step GD training: when the step size exceeds the classical stability threshold (step size × sharpness < 2), the loss does not diverge; instead, sharpness fluctuates around the critical value 2/step size, and training oscillates near the stability edge while still converging. This phenomenon has been observed in standard training of fully connected networks and Transformers. This paper finds that sharpness evolution in AT exhibits two-phase dynamics, different from the single edge-of-stability behavior in ST, indicating that adversarial training shapes the loss surface in a fundamentally different way. Sharpness changes cannot be explained solely within the ST framework and require sharpness-related regularization or early stopping strategies designed specifically for AT.

Summary

This study systematically characterizes the GD dynamics of large-step AT and proposes three differences from ST: the implicit bias gap persists, there is no unified convergence rate, and sharpness exhibits two-phase dynamics. These results advance the theoretical understanding of adversarial training behavior under the large step sizes commonly used in practice, and provide new perspectives for AT tuning and robustness analysis.

One-line Summary

Gradient descent dynamics in large-step adversarial training differ fundamentally from standard training in three key ways.

Paragraph Summary

This paper studies gradient descent dynamics in adversarial training (AT) with large constant step sizes, revealing three key differences from standard training (ST): first, the implicit bias gap between AT and ST, already established under small step sizes, is shown to persist under arbitrarily large constant step sizes, indicating that implicit bias is an intrinsic property of AT rather than an artifact of the small-step-size limit; second, no unified convergence rate for optimization error exists in AT, because inner maximization makes the optimization landscape more complex and convergence behavior dependent on the specific problem instance; third, loss sharpness in AT during neural network training can exhibit two-phase dynamics, different from the edge-of-stability behavior in ST. The study covers implicit bias and optimization analysis in adversarial linear logistic regression as well as sharpness evolution in neural networks, providing a new theoretical perspective for AT tuning and robustness analysis under the large step sizes commonly used in practice.