Rank-Aware Metric
A class of evaluation metrics that examines not only the absolute probability change of a target token but also its relative rank change within the candidate vocabulary, used to distinguish precise directional suppression from artifacts of overall perturbation magnitude.
Timeline (last 90 days)
Rank-aware metrics not only examine whether the absolute probability or log-probability of the target token decreases, but also examine the change in the token's relative ranking in the candidate vocabulary.
For practitioners, reporting only a logprob decrease is insufficient to support the conclusion that SAE features effectively suppress PII; at minimum, direction-matched random controls and rank-aware metrics should be added.
All Facts (2)
Rank-aware metrics not only examine whether the absolute probability or log-probability of the target token decreases, but also examine the change in the token's relative ranking in the candidate vocabulary.
50%UnverifiedFor practitioners, reporting only a logprob decrease is insufficient to support the conclusion that SAE features effectively suppress PII; at minimum, direction-matched random controls and rank-aware metrics should be added.
50%