Better Metrics Won't Save You: What You Compare Matters More Than How You Compare It

The Hidden Assumption Behind Representational Similarity Metrics

At the intersection of deep learning and cognitive neuroscience, representational similarity analysis has become a core tool for comparing internal representations across models and systems. Whether it's the classic RSA (Representational Similarity Analysis), CKA (Centered Kernel Alignment), or Procrustes analysis, each method has its advocates, and researchers typically invest considerable effort in selecting the "most appropriate" metric.

But a new study raises a question that has been widely overlooked: the differences between these metrics matter far less than what you choose to compare. In other words, while researchers debate which similarity measure is more scientifically sound, what actually drives the conclusions is how the data is organized.

Where Invariance Properties Fall Short

Every representational similarity metric is essentially defined by the transformations it treats as irrelevant. RSA, CKA, and Procrustes each have different tolerances for rotation, scaling, and linear transformations — these are their invariance properties, and researchers typically use them to judge whether a given metric is appropriate for their use case.

However, all of these metrics share a common hidden assumption: they accept the "set of conditions" as a given. That set is fixed by two decisions — which inputs are included, and how those inputs are grouped. Crucially, both of these decisions happen before the matrix the metric operates on even exists.

This means that no matter which metric you choose, no matter how well-designed its invariance properties are, those properties cannot reach back and constrain the choices made during data organization. The rigor of the metric becomes meaningless in the face of arbitrariness in data construction.

To be more specific: RSA compresses representations into a pairwise dissimilarity matrix (RDM, Representational Dissimilarity Matrix) and is invariant to any monotonic transformation; CKA measures similarity through centered alignment of kernel matrices and is invariant to orthogonal transformations and isotropic scaling; Procrustes analysis aligns two sets of representations via optimal rotation and is invariant to rotations and reflections. These invariances may seem meaningfully distinct, but regardless of the method, the input is always a "conditions × features" matrix — the set of activation vectors obtained by passing a pre-selected group of stimuli through two systems. The rows of that matrix (the conditions) are determined by the researcher at the experimental design stage. The metric has no awareness of this, and no ability to correct for it.

Testing with the Acoustic Organization of Vowels

Verifying this argument requires a scenario where the "ground truth" is known independently of the systems being compared. The researchers chose the acoustic organization of vowels as their testing platform, spanning fifteen languages and twenty-seven representations.

This was a clever choice. The organization of vowels in acoustic space has an objective basis in linguistics and acoustics that is independent of whatever model representations are being compared. This provides an external reference frame, allowing researchers to genuinely measure how much different choices affect the outcome.

The acoustic structure of vowels is typically described using formant frequencies (F1, F2): F1 correlates with vowel height and F2 with frontness/backness. Vowel systems across different languages exhibit relatively stable distributions in this two-dimensional acoustic space, and these distributions are independent of any computational model or neural measurement. This makes vowels a rare test case with an external ground truth — researchers can use the same linguistically established structure to examine how much condition selection perturbs the conclusions of similarity analysis, without worrying that the ground truth itself shifts with the choice of conditions. Covering fifteen languages further ensures the findings don't depend on any single language's idiosyncrasies.

What You Compare Matters Far More Than How Many

The study's central finding is striking: with category count held constant, changing which categories are compared shifts the gap between two representations by more than twice the size of the effect being measured; yet changing how many categories are compared has almost no effect at all.

This contrast clearly exposes the problem. Researchers often agonize over sample size and number of conditions, assuming that more conditions yield more robust conclusions. But the data show that what determines the conclusion is not how many things you compare, but which things you choose to compare. Category selection is itself a variable capable of reversing rankings entirely.

Model Rankings Are Identifiable; Language Rankings Are Not

Further analysis distinguishes between the robustness of two types of conclusions. Under these various choices, model rankings remain identifiable — in other words, conclusions about which model is better hold up even as condition selection varies.

But cross-language rankings are not identifiable. When trying to compare the similarity rankings of representations across different languages, the conclusions can be completely reversed simply by changing category selection. This is an important warning: the same methodology can yield reliable conclusions along one dimension while being entirely unreliable along another.

"Identifiability" here is a statistical concept referring to whether a conclusion can be robustly reproduced under reasonable variation in the data. Model rankings being identifiable means that even when researchers substitute different subsets of conditions, the relative ordering of models on the dimension of "similarity to the target representation" remains largely consistent — the conclusions are reasonably robust. Cross-language rankings being non-identifiable means that conclusions about "which language's representations are more similar to the target" are highly sensitive to condition selection, and different category subsets may produce mutually contradictory orderings. This asymmetry signals to researchers that the same methodology may differ fundamentally in reliability across different comparison dimensions — experience from one type of problem cannot be directly extrapolated to another.

Implications for Research Practice

The significance of this work extends well beyond vowel research. It sends a warning to the entire community using representational similarity analysis: when we report that one model is closer to the brain than another, or that one representation is superior, we must examine whether that conclusion is simply an artifact of how the data was organized.

Switching to a better metric cannot rescue an analysis that has arbitrariness baked in at the data construction stage. What genuinely needs to be explicitly reported and justified are the two upstream decisions: "which inputs are included" and "how they are grouped." In many cases, these decisions are treated as self-evident and never seriously discussed in published papers.

For AI interpretability research, cognitive modeling, and any work that relies on representational comparisons, this calls for a methodological rethink: first confirm that your condition set is robust, then worry about which similarity metric to use. Otherwise, a sophisticated metric only dresses up an unreliable conclusion in the appearance of rigor.