Does a Larger Batch Actually Learn Slower? The RLVR "Two-Clock" Framework Reveals the Truth About Batch Size and Learning Rate Scaling

In the training practice of Reinforcement Learning with Verifiable Rewards (RLVR), a widely adopted intuition is that increasing the rollout batch size while scaling the learning rate by the square-root rule can accelerate training while preserving "per-sample progress." But a new theoretical study challenges this rule of thumb—a larger batch may not only learn more slowly, but also hit performance collapse at a predictable point in time.

The Two-Clock Framework: Separating "Progress" from "Collapse"

RLVR relies on expensive model-generated responses, which directly gives rise to the combination of "larger rollout batch + square-root learning rate scaling." Under the idealized assumption of a single-step optimizer, the square-root rule was reported to roughly preserve per-sample progress. But once it enters actual multi-step training, the behavior of this rule becomes hard to pin down.

The researchers propose a "two-clock" framework that splits the training process into two independent timelines: one measuring "how many samples are needed to make progress," and the other measuring "how many samples are consumed before the model collapses." The key insight of this split is that the square-root rule only preserves the per-sample first-order drift, but does not preserve true "progress."

Under a local quadratic model, a larger batch means fewer but longer update steps, and each step incurs greater loss on the curvature. This leads to the existence of a learning-rate-dependent critical batch size, beyond which sample efficiency declines.

RLVR (Reinforcement Learning with Verifiable Rewards) refers to a reinforcement learning paradigm in which the reward signal comes from programmatically verifiable answer correctness (such as the standard answer to a math problem, or whether code passes its unit tests). Compared with RLHF, which relies on human preferences or reward model scores, RLVR's reward determination has almost zero noise and zero cost, and is therefore widely used in tasks such as mathematical reasoning and code generation. Its cost is that the reward is sparse and binary—positive feedback is only obtained when the entire reasoning chain ultimately leads to the correct answer, which forces training to rely on a large amount of rollout sampling to obtain effective gradient signals, and also makes the computational overhead of rollouts the bottleneck of the entire training pipeline. It is precisely this cost structure that gave rise to the engineering intuition of "using a larger batch to amortize generation overhead, and scaling the learning rate accordingly."

The square-root scaling rule is an empirical rule originating from stochastic optimization: when the batch size increases by a factor of $k$, multiply the learning rate by $\sqrt{k}$. Its theoretical basis is that in gradient descent, the parameter variance caused by mini-batch noise is proportional to the square of the learning rate, while the noise magnitude shrinks by $1/\sqrt{k}$ as the batch grows. Therefore, scaling the learning rate by $\sqrt{k}$ exactly maintains the signal-to-noise ratio of the gradient. Under the idealized model of a single-step update, this is believed to keep "per-sample progress" unchanged, that is, to exchange a larger batch for equal progress but fewer update steps. The core challenge of this paper is aimed precisely at this premise: the rule only anchors the first-order drift, and does not anchor the true convergence speed accumulated over multi-step iterations, nor does it constrain how far the model drifts away from the initial region during long training—the latter is the key quantity that triggers performance collapse.

The Drift-Threshold Hypothesis: Why the Failure Clock Is Approximately Invariant

If the progress clock slows down as the batch grows, what about the failure clock? The research team proposes the "drift-threshold hypothesis," predicting that there exists a complementary failure clock whose sample count remains approximately unchanged under the square-root rule.

In other words, when you increase the batch along the trajectory of the square-root rule, the model's rate of progress slows down, but the number of samples required to trigger collapse hardly moves. The misalignment of the two clocks is precisely the root of the counterintuitive phenomenon that "a larger batch learns more slowly but still fails on schedule."

This hypothesis is not mere speculation on paper. The researchers conducted controlled GRPO experiments on Qwen2.5-Math-1.5B with batch sizes spanning a factor of 16, and the results supported both predictions above.

Ratio clipping is a stabilization mechanism in policy optimization algorithms such as PPO/GRPO. By comparing the probability ratio of the new and old policies on the same sample, it constrains this ratio within the interval $[1-\epsilon, 1+\epsilon]$ (with $\epsilon=0.2$ commonly used in GRPO), thereby limiting the magnitude of change that a single update can make to the policy distribution. Its theoretical motivation comes from trust-region methods: an overly aggressive policy update causes the importance sampling ratio to deviate too far from 1, causing the variance of the off-policy estimate to explode and training to diverge. A side effect of clipping is that once a sample's ratio is truncated, its gradient contribution no longer reflects the true direction of policy improvement, which may cause optimization stagnation. In this paper, "relaxing clipping in exchange for a brief speedup" is exactly a manifestation of this trade-off: the wider the boundary, the greater the distance that can be moved in a single step, the faster the short-term progress, but the more rapidly accumulating drift also causes one to hit the failure clock earlier.

Experimental Evidence: Validation Across a 16x Range of Batch Sizes

The experimental results yield several noteworthy findings:

Both predictions are supported. Across a 16x range of batch sizes, both the progress gap and the approximate invariance of the failure clock appear as predicted by theory.

The role of ratio clipping is limited. The clipping mechanism can indeed delay failure, with an especially pronounced effect for larger batches, but without it the progress gap still exists. This shows that clipping is more like a "delay agent" than a fundamental solution.

Relaxing clipping is a double-edged sword. Widening the clipping boundary trades stability for a brief period of acceleration. This speed benefit is temporary and does not solve the fundamental problem in the long run.

Reproducible across settings. The progress gap reappears on another dataset, on a general-purpose checkpoint, and on a 7B model, indicating that this is not an accidental phenomenon of some specific configuration.

Practical Implications: A Multi-Step Scheduling Strategy That Starts Small and Then Goes Large

Based on the two-clock framework, the researchers derive a simple and practical multi-step RLVR scheduling scheme: start training below the critical batch size, and switch to a larger batch before failure arrives.

The idea behind this strategy is straightforward—use the small-batch phase to obtain better sample efficiency, and proactively upgrade to a large batch before approaching the collapse point in order to delay failure. The code has been open-sourced on GitHub (WinstonLiyt/two-clocks-rlvr).

For teams currently doing RLVR training, this means that blindly applying "large batch + square-root scaling" may not be worthwhile. Understanding the relative positions of the progress clock and the failure clock is more valuable than simply pursuing a larger batch.

One-line Summary

A larger batch plus square-root scaling simultaneously slows progress while leaving the collapse point unchanged.

Paragraph Summary

This study challenges the rule of thumb in RLVR of "increasing the rollout batch and scaling the learning rate by the square-root rule." The authors propose a two-clock framework that decomposes training into a "progress clock," measuring the number of samples needed to make progress, and a "failure clock," measuring the number of samples that can be consumed before collapse. Under a local quadratic model, a larger batch means fewer but longer update steps, with greater curvature loss per step, so there exists a learning-rate-dependent critical batch size beyond which sample efficiency declines; meanwhile, the drift-threshold hypothesis predicts that the failure clock is approximately invariant under the square-root rule. GRPO experiments on Qwen2.5-Math-1.5B across a 16x range of batch sizes support both predictions and show that ratio clipping can only delay failure, not eliminate the progress gap. On this basis, the authors propose a multi-step scheduling strategy of "small batch first, large batch later."