Bregman Divergence
A family of generalized distance measures proposed by mathematician Lev Bregman in 1967. Defined on the basis of strictly convex functions, the family encompasses common measures such as Euclidean distance, KL divergence, and Itakura-Saito divergence, and has deep theoretical connections to exponential family distributions.
Timeline (last 90 days)
Online mirror descent (OMD) uses the Bregman divergence as the distance measure, and for convex Lipschitz-continuous loss sequences it guarantees an upper bound on cumulative regret of O(√T) or O(log T).
The paper connects three mathematical tools—contrastive learning, exponential family distributions, and information geometry—to establish a rigorous mapping between probability distributions over anchor points and Bregman geometry over the representation space.
The Bregman divergence was proposed by mathematician Lev Bregman in 1967.
When the convex function φ in the Bregman divergence is taken as the squared norm, it yields the squared Euclidean distance; when taken as the negative entropy, it yields the KL divergence; and when taken as the negative logarithm, it corresponds to the Itakura-Saito divergence.
The log-normalizing function of the exponential family distribution can serve as the convex function φ in the definition of the Bregman divergence, establishing a one-to-one correspondence between the anchor point probability distribution model and the geometric metric of the representation space.
All Facts (5)
Online mirror descent (OMD) uses the Bregman divergence as the distance measure, and for convex Lipschitz-continuous loss sequences it guarantees an upper bound on cumulative regret of O(√T) or O(log T).
50%UnverifiedThe paper connects three mathematical tools—contrastive learning, exponential family distributions, and information geometry—to establish a rigorous mapping between probability distributions over anchor points and Bregman geometry over the representation space.
50%UnverifiedThe Bregman divergence was proposed by mathematician Lev Bregman in 1967.
50%UnverifiedWhen the convex function φ in the Bregman divergence is taken as the squared norm, it yields the squared Euclidean distance; when taken as the negative entropy, it yields the KL divergence; and when taken as the negative logarithm, it corresponds to the Itakura-Saito divergence.
50%UnverifiedThe log-normalizing function of the exponential family distribution can serve as the convex function φ in the definition of the Bregman divergence, establishing a one-to-one correspondence between the anchor point probability distribution model and the geometric metric of the representation space.
50%