Unverified60% confidenceFactExact time
Ethayarajh et al. revealed through empirical research in 2019 that word vectors in pre-trained models such as BERT and GPT are highly concentrated in a narrow cone-shaped region, exhibiting anisotropy.
2
Sources
60%
Confidence
Long-term
Relevance
7/7/2026
First Seen
Sources
分散损失：破解小语言模型嵌入坍缩的关键技术
hackernewshackernews7/3/2026
Related Claims
UnverifiedVectors produced by many pre-trained models are not evenly distributed in space but are instead clustered in a narrow cone-shaped region, a phenomenon known as anisotropy.68% similarUnverifiedMetrics based on pretrained embeddings, such as BERTScore, attempt to introduce semantic similarity but still operate at the sentence level and struggle to track whether specific factual claims in an answer are consistent with the context.65% similarUnverifiedContext confusion produces narrow misalignment, where misalignment is concentrated in specific domains semantically close to the training data, rather than a broad degradation of the model's overall behavior.65% similarUnverifiedWhen there is a pronounced heteroscedastic structure in the data, a single global early-stopping point may not be optimal, and region-adaptive training strategies are worth exploring.65% similarUnverifiedOver-reliance on the fine-tuned part during fine-tuning may be precisely the root cause of rare-class capabilities being obscured; directing the model's computation moderately toward the pretrained components may be a more robust fine-tuning route.63% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/189248API
curl https://kongchang.com/api/v1/knowledge/claims/189248MCP
get_claim(id=189248)