Glassy State
A phase of the attention mechanism in which the free-energy landscape contains many similar local minima, so retrieval falls into incorrect patterns. It manifests as sensitivity to small input perturbations, unstable outputs, and large variation in results across random seeds, making inference results unreliable.
Timeline (last 90 days)
The research identifies the existence of a first-order phase transition in attention mechanisms, where the system can switch between two截然 different phases: a glassy state and a robust inference state.
In the glassy state, attention inference is unstable, and the system may fall into a large number of local minima, leading to unreliable inference results.
Understanding the critical threshold of the phase transition helps in designing more stable attention mechanisms and avoiding falling into the glassy state.
All Facts (3)
The research identifies the existence of a first-order phase transition in attention mechanisms, where the system can switch between two截然 different phases: a glassy state and a robust inference state.
50%UnverifiedIn the glassy state, attention inference is unstable, and the system may fall into a large number of local minima, leading to unreliable inference results.
50%UnverifiedUnderstanding the critical threshold of the phase transition helps in designing more stable attention mechanisms and avoiding falling into the glassy state.
50%