Anisotropy
A phenomenon describing the uneven distribution of embedding vectors in high-dimensional space, in which vectors produced by pre-trained language models tend to cluster within a narrow cone rather than spreading uniformly. This causes artificially high cosine similarity scores between random texts, reducing the reliability of retrieval systems.
Timeline (last 90 days)
The vast majority of embedding models are anisotropic—their vectors are not uniformly distributed across a hypersphere but are instead clustered within a narrow cone, causing the cosine similarity of unrelated texts to be inflated.
The anisotropy problem existed in the era of static word vectors and was particularly severe for GloVe embeddings; it was somewhat alleviated but not eliminated with the advent of the Transformer era due to contextual modeling.
The root cause of anisotropy lies in the training objectives and architectural design of pre-trained language models (such as the BERT series), which tend to concentrate high-frequency words and general semantics in certain specific directions.
Vectors produced by many pre-trained models are not evenly distributed in space but are instead clustered in a narrow cone-shaped region, a phenomenon known as anisotropy.
Even in modern embedding models optimized through instruction fine-tuning or contrastive learning (such as BGE and E5), the anisotropy problem has not been completely eliminated, only reduced in degree.
Academics use Effective Rank to quantify the anisotropy phenomenon; an ideal isotropic embedding space should make full use of all dimensions.
If the cosine similarity between a large number of random, unrelated texts is consistently high (e.g., concentrated above 0.6), it indicates significant anisotropy in the embedding space.
All Facts (7)
The vast majority of embedding models are anisotropic—their vectors are not uniformly distributed across a hypersphere but are instead clustered within a narrow cone, causing the cosine similarity of unrelated texts to be inflated.
50%UnverifiedThe anisotropy problem existed in the era of static word vectors and was particularly severe for GloVe embeddings; it was somewhat alleviated but not eliminated with the advent of the Transformer era due to contextual modeling.
50%UnverifiedThe root cause of anisotropy lies in the training objectives and architectural design of pre-trained language models (such as the BERT series), which tend to concentrate high-frequency words and general semantics in certain specific directions.
50%UnverifiedVectors produced by many pre-trained models are not evenly distributed in space but are instead clustered in a narrow cone-shaped region, a phenomenon known as anisotropy.
50%UnverifiedEven in modern embedding models optimized through instruction fine-tuning or contrastive learning (such as BGE and E5), the anisotropy problem has not been completely eliminated, only reduced in degree.
50%UnverifiedAcademics use Effective Rank to quantify the anisotropy phenomenon; an ideal isotropic embedding space should make full use of all dimensions.
50%UnverifiedIf the cosine similarity between a large number of random, unrelated texts is consistently high (e.g., concentrated above 0.6), it indicates significant anisotropy in the embedding space.
50%