The Silent Killer in Long-Horizon RL: Numerical Drift Between Sampling and Scoring Engines

The Overlooked Numerical Bug

In the engineering practice of large language model reinforcement learning (RL), the most persistent problems are often not algorithmic — they're hidden numerical consistency issues lurking beneath the surface. A post shared on Twitter cuts right to the heart of this pain point: the most intractable bugs in long-horizon RL are numerical in nature. When the engine responsible for generating rollouts (trajectory sampling) and the engine responsible for scoring assign different probabilities to the same batch of tokens, training quietly begins to drift.

This drift doesn't immediately crash training. Instead, it acts like a chronic illness, gradually eroding the reliability of the training signal until model behavior diverges from expectations. For tasks requiring reward optimization over long sequences, this class of problem is especially devastating.

Why Probability Inconsistencies Arise

In a typical RLHF or RL training pipeline, there are usually two (or more) distinct execution paths: an inference/sampling engine responsible for efficiently generating model rollouts, and a training/scoring engine responsible for computing log probabilities, advantage functions, or rewards over those outputs.

Ideally, feeding the same token sequence into both engines should yield identical probability distributions. In practice, however, the two can diverge at several points:

• Tokenization differences: If the tokenization logic on the sampling side isn't perfectly aligned with the scoring side, the same text may be split into different token sequences, causing probabilities to mismatch.
• Numerical precision differences: Subtle differences in floating-point computation, kernel implementations, and batching strategies between engines can accumulate into significant errors over long sequences.
• Distortions from inference optimizations: Sampling engines often employ speed optimizations such as quantization and operator fusion, which can alter the precise values of output probabilities.

In short-sequence tasks, these tiny discrepancies may be tolerable. But in long-horizon RL, errors amplify along the time dimension, distorting the training signal.

This issue is especially pronounced in heterogeneous architectures that use dedicated inference engines like vLLM or TensorRT-LLM for rollout generation and PyTorch/DeepSpeed for training. Take PPO (Proximal Policy Optimization) as an example: the algorithm needs to compute the ratio of "old policy probabilities" to "new policy probabilities" (the importance ratio). Any small divergence between the two engines directly contaminates this ratio. When the importance ratio exhibits systematic bias, PPO's clipping mechanism will incorrectly truncate legitimate gradient updates or allow anomalous updates to pass through unchecked — ultimately manifesting as inexplicable oscillations in the training curve or stalled reward improvement. KV cache reuse strategies can also introduce hidden risks: cached intermediate activations may produce slightly different logit outputs when batch sizes or sequence concatenation patterns change. This is nearly imperceptible in a single inference pass, but accumulates persistently across the many iterations of long-horizon RL.

Aligning Tokenization Is the Critical Step

The solution proposed is clear: keeping tokenization consistent across both engines is the core requirement for trustworthy training signals.

This may seem like a minor engineering detail, but it directly affects the correctness of the entire RL training process. If rollouts are generated using one tokenization scheme but token boundaries shift during scoring, the model's "own outputs" as seen during generation become misaligned with the actual targets being scored. Gradients computed from this misalignment are, in essence, optimizing the wrong objective.

Aligning tokenization means:

1. Ensuring the sampling side and scoring side share the exact same tokenizer version and configuration;
2. Maintaining strict consistency in special token handling, boundary processing, and encode/decode round-trips;
3. Establishing a verification mechanism to regularly compare tokenization results and probability outputs from both sides on identical inputs.

Round-trip consistency in tokenization is a validation dimension that is frequently overlooked: encoding text into token IDs and then decoding back to text does not always reproduce the original string exactly. When a token sequence generated on the sampling side is decoded and then re-encoded — for example, during storage, transmission, or prompt concatenation — different token boundaries may result. This is especially true for Chinese, Japanese, and code, where a single semantic unit may have multiple valid tokenizations. Special tokens such as <|im_start|> and <eos> are also handled differently across frameworks; some inference engines automatically strip or append these tokens, and if the training framework doesn't apply the same treatment, length mismatches are introduced. A recommended practice is to add assertion checks in the pipeline: for any given rollout, verify that the token sequence remains identical through the full cycle of "sampling-side input token IDs → decode → scoring-side re-encode," and incorporate this check as a standard health check in the data pipeline.

Implications for RL Engineering Practice

This insight has direct practical value for teams building large-model RL systems. When training exhibits unexplained performance degradation, rather than suspecting algorithmic hyperparameters first, teams should prioritize investigating numerical consistency between the sampling and scoring engines.

A practical diagnostic approach: take a fixed batch of token sequences, feed them into both engines, and compare the per-token probabilities they output. If systematic divergence is found, the problem typically lies in tokenization or numerical implementation — not in the algorithm logic.

In an era where training efficiency is paramount and heterogeneous inference engines are widely used, maintaining "numerical trustworthiness" as data flows between different engines is becoming an invisible threshold for deploying long-horizon RL in practice. Teams that cleanly resolve these low-level consistency issues will have more reliable training signals.

"Numerical consistency testing" can be systematized into a regression test suite rather than remaining an ad hoc diagnostic tool. Concretely, this involves: building a fixed test set that includes edge cases (very long sequences, outputs with special characters, mixed multilingual text); running it automatically after every engine version upgrade or configuration change; computing the maximum absolute error and KL divergence of per-token log probabilities between both sides; setting a tolerance threshold (e.g., KL < 1e-4) and blocking training when it is exceeded. This "numerical CI/CD" philosophy has already been put into practice in the RL infrastructure of several leading labs — it essentially transforms hidden numerical risk into a quantifiable, monitorable engineering metric. For teams using distributed training strategies such as FSDP or tensor parallelism, an additional verification step is needed: confirming that logit outputs for the same sequence are deterministic across different parallelism configurations, since the floating-point accumulation order of communication operators can change depending on how parallelism is set up.