Reverse KL Divergence
A divergence measure used in variational inference to quantify the difference between an approximate distribution and a target distribution. It has a zero-forcing property, which causes the approximate distribution to preferentially cover a few high-probability modes of the target distribution, leading to mode collapse.
Timeline (last 90 days)
KL divergence is defined as D_KL(P||Q)=∫log(dP/dQ)dP, and the reverse KL divergence swaps the positions of P and Q, i.e., D_KL(Q||P).
The reverse KL divergence is zero-forcing and tends toward mode seeking, causing severe mode collapse when training diffusion samplers with variational inference.
SeTPO introduces teacher signals only for low-reward trajectories, uses reverse KL divergence to derive token-level teacher feedback, and incorporates it into the policy advantage function.
Reverse KL divergence is mode-seeking, while forward KL divergence is mean-seeking.
All Facts (4)
KL divergence is defined as D_KL(P||Q)=∫log(dP/dQ)dP, and the reverse KL divergence swaps the positions of P and Q, i.e., D_KL(Q||P).
50%UnverifiedThe reverse KL divergence is zero-forcing and tends toward mode seeking, causing severe mode collapse when training diffusion samplers with variational inference.
50%UnverifiedSeTPO introduces teacher signals only for low-reward trajectories, uses reverse KL divergence to derive token-level teacher feedback, and incorporates it into the policy advantage function.
50%UnverifiedReverse KL divergence is mode-seeking, while forward KL divergence is mean-seeking.
50%