Smaller models are more prone to embedding collapse than larger models, due to reasons including limited parameter capacity, a tendency toward optimization shortcuts, and amplified anisotropy issues.