KL Divergence

The RLHF pipeline consists of three steps: training a reward model to predict human preference scores, adjusting language model parameters using the PPO reinforcement learning algorithm, and using KL divergence constraints to prevent the model from deviating from its original language capabilities.

Common methods for detecting data drift include the KS test, PSI (Population Stability Index), and sliding-window-based statistical monitoring.

Cross-entropy = entropy of the true distribution + KL divergence.

从KL散度角度看，每一代合成数据与真实分布之间的散度都在单调递增，且误差会在迭代中累积

KL divergence is asymmetric, i.e., D(p,q) ≠ D(q,p).

In token-level sequence generation tasks, the KL constraint is computed token by token, so tokens at different positions are subject to different constraint strengths.

Forward KL tends to make the student policy cover all modes of the teacher policy, learning comprehensively but possibly not precisely enough; reverse KL tends to make the student policy concentrate on the high-probability regions of the teacher policy, learning precisely but possibly missing certain modes.

Lorden's lower bound shows that, under a given constraint on the average time to false alarm, the detection delay is at least proportional to 1/D_KL.

Under a fixed threshold and incremental scaling, the detection delay of DI-SCUSUM is inversely proportional to the KL divergence.

Under the assumption that observations follow a fixed empirical distribution, the average post-change increment of DI-SCUSUM is proportional to the KL divergence from the smoothed post-change distribution to the smoothed pre-change distribution.

Drift detection algorithms generally monitor changes through statistical tests (such as ADWIN and the Page-Hinkley test) or distribution distance metrics (such as KL divergence and Wasserstein distance).

Dreamer V1's loss function consists of a weighted sum of a reconstruction term, a KL divergence term, and a reward prediction term: the reconstruction term usually uses mean squared error or negative log-likelihood, and the reward prediction term uses cross-entropy or MSE.

In Dreamer V1, the KL divergence term is used to constrain the distance between the posterior distribution q(z_t|h_t,o_t) and the prior distribution p(z_t|h_t); if the KL weight is too large, the posterior is forced to approach the prior, causing the latent variables to lose observational information (information loss), and if the weight is too small, the transition model cannot generate meaningful imagined trajectories from the prior (posterior collapse).

Under the assumption of smooth conditional density models, for any β>0, selecting an appropriate sampling grid allows the average conditional KL divergence to achieve a convergence rate of Õ_p(n^(−2β/(d_x+d_y+2β))).

KL divergence and TV distance are related through Pinsker's inequality: TV(p,q) ≤ √(KL(p‖q)/2).

The scheduling-averaged divergence first computes the KL divergence under each schedule and then takes the expectation, i.e., E_U D_KL(p||q_U).

Jensen's inequality guarantees that D_KL(p||E_U q_U) ≤ E_U D_KL(p||q_U), meaning that mixing first and then measuring is always no greater than measuring first and then averaging.

KL divergence is asymmetric, D_KL(p||q) ≠ D_KL(q||p), and non-negative, equaling zero if and only if the two distributions are identical.

During training, REMORY uses KL divergence to measure the difference in output distributions between a frozen LLM seeing the full history and seeing the summary plus residual tokens.

The 2-Wasserstein distance is more sensitive to the geometric shape of the support sets of distributions than KL divergence, and can yield meaningful finite distance values even when the support sets of two distributions do not overlap.