CKA
A representational similarity metric for neural networks proposed by Kornblith et al. in 2019. Invariant to linear transformations, CKA enables cross-model and cross-layer comparison of representations and is widely used in neural network interpretability research.
Timeline (last 90 days)
The paper points out that calibration causes neighborhood overlap to continue to grow as model scale increases, yet essentially eliminates the growth of centered kernel alignment (CKA).
CKA measures the overall similarity of two representation spaces, first centering the features and then comparing the correspondence between the sample similarity matrices, and is insensitive to global linear transformations.
CKA measures similarity through centered alignment of kernel matrices, remaining invariant to orthogonal transformations and isotropic scaling.
All Facts (3)
The paper points out that calibration causes neighborhood overlap to continue to grow as model scale increases, yet essentially eliminates the growth of centered kernel alignment (CKA).
50%UnverifiedCKA measures the overall similarity of two representation spaces, first centering the features and then comparing the correspondence between the sample similarity matrices, and is insensitive to global linear transformations.
50%UnverifiedCKA measures similarity through centered alignment of kernel matrices, remaining invariant to orthogonal transformations and isotropic scaling.
50%