Information Bottleneck
An information-theoretic framework proposed by Tishby et al. in 2000, asserting that deep networks should compress input information while retaining the minimal sufficient statistics relevant to the task, thereby achieving an optimal balance between generalization capability and expressive power.
Timeline (last 90 days)
Context compression typically conveys the information of the original long text through a capacity-limited intermediate representation (such as a fixed-length summary or latent vector), a process known as the information bottleneck.
The idea of action predictability is akin to the information bottleneck principle in contrastive learning: when a model needs to complete a certain prediction task, the encoder is forced to retain the minimal sufficient information necessary to complete the task.
All Facts (2)
Context compression typically conveys the information of the original long text through a capacity-limited intermediate representation (such as a fixed-length summary or latent vector), a process known as the information bottleneck.
50%UnverifiedThe idea of action predictability is akin to the information bottleneck principle in contrastive learning: when a model needs to complete a certain prediction task, the encoder is forced to retain the minimal sufficient information necessary to complete the task.
50%