Rényi Concentration Metrics: Freeing MLP Interpretability Measures from Network-Scale Dependence

Multilayer perceptrons (MLPs), one of the most classic neural network architectures, have long posed a challenge when it comes to the interpretability of their internal representations. The Kolmogorov-Arnold Geometry (KAG) framework reveals the internal computational structure of a network by analyzing the local response of hidden layers to input changes—that is, the minors of the Jacobian matrix. A recent paper points out a fundamental flaw in the existing metric for measuring minor concentration, the Participation Ratio (PR): its maximum value depends on network scale, making it impossible to judge, independent of architecture, whether a given PR value represents high or low concentration. To address this, the authors propose a class of α-RC metrics based on Rényi divergence, which normalize the concentration measure to the [0,1] interval and are independent of network architecture.

The KAG Framework and the Limitations of the PR Metric

The core idea of Kolmogorov-Arnold Geometry is that, for a single-hidden-layer MLP, the local response of the hidden layer to the input can be described by the minors of the Jacobian matrix. The relative magnitudes of these minors determine the "concentration" of the network's local behavior—if a few minors dominate the overall response, the network relies on a few key paths in that region; otherwise, the response is dispersed.

Previously, concentration was always measured using the Participation Ratio. Although PR has some favorable properties, its maximum value varies with network scale (the number of minors). This means that a PR value alone cannot be interpreted as "high concentration" or "low concentration"; it must be considered together with the specific architecture. This scale dependence limits PR's potential as a general-purpose interpretability metric.

Minors of the Jacobian matrix refer to the determinants of square matrices formed by selecting several rows and several columns from the Jacobian matrix. In the KAG framework, the local response of a single-hidden-layer MLP's hidden layer to the input can be described by the Jacobian matrix, where each element represents the partial derivative of a hidden unit's output with respect to an input dimension. The minors capture the joint local response strength between multiple hidden units and multiple input dimensions. The larger the absolute value of a minor, the more significant the response in that subspace direction. By analyzing the relative magnitudes of all minors, one can determine whether the network relies on a few key paths locally (high concentration) or responds uniformly (low concentration). The number of minors grows combinatorially with network scale (input dimension and number of hidden units), which is also the root cause of the PR metric's scale dependence.

The α-RC Metric: A Normalization Scheme Based on Rényi Divergence

The family of α-RC metrics proposed in the paper quantifies the degree of deviation from a uniform distribution by comparing normalized minor magnitudes with a uniform distribution, using Rényi divergence (of order α). This design brings several key properties:

• Boundedness: Regardless of network architecture, all α-RC metrics fall within the [0,1] interval. A value of 0 indicates that magnitudes are uniformly distributed across all minors, while 1 indicates that the magnitude is entirely carried by a single minor.
• Monotonicity: The metric value is non-decreasing with α, providing concentration perspectives of different "sharpness."
• Explicit connection to PR: When α=2, α-RC reduces to an explicit transformation of the Participation Ratio, thereby eliminating scale dependence while preserving PR's interpretability.

These properties allow α-RC metrics to directly compare concentration across networks of different scales without worrying about architecture-induced bias.

Rényi divergence is a family of measures of the difference between probability distributions, proposed by Alfréd Rényi and parameterized by an order α. As α→1, Rényi divergence reduces to KL divergence; when α=2, it is related to χ² divergence; as α→∞, it becomes more sensitive to the maximum difference between distributions. In the α-RC metric, the authors treat the normalized minor magnitudes as a discrete distribution and compute its Rényi divergence from the uniform distribution. Through an appropriate transformation, this divergence is mapped to the [0,1] interval, yielding a bounded concentration metric. The α parameter controls sensitivity to "sharpness": smaller α makes the metric more sensitive to moderate concentration, while larger α emphasizes extreme concentration. This design enables α-RC to flexibly capture concentration patterns at different levels while maintaining architecture-independent normalization.

Computational Experiments and Empirical Value

The authors trained single-hidden-layer MLPs on synthetic datasets and validated the effectiveness of the α-RC metric through computational experiments. Experiments show that the new metric more clearly reflects the degree of concentration of the network's internal representations and is unaffected by network scale. This provides a more reliable measurement tool for future research and is expected to advance the practical application of the KAG framework in model interpretability analysis.

Although the paper's experimental scale is limited (only synthetic datasets and a single hidden layer), its theoretical contribution is clear: the α-RC metric solves the scale-dependence problem of PR and provides an architecture-independent concentration measure for MLP interpretability analysis. For researchers and engineers concerned with the internal mechanisms of neural networks, this metric is worth adding to the interpretability toolbox.

Synthetic datasets are often used in neural network interpretability research to control variables and validate theoretical metrics. The authors likely generated data with known structure (such as low-rank, sparse, or uniform) and trained single-hidden-layer MLPs to precisely compute Jacobian minors and evaluate the behavior of the α-RC metric. The advantage of synthetic data is that network scale, input dimension, and number of hidden units can be systematically varied to test whether the metric is truly architecture-independent. Although experiments on real data have not yet been conducted, validation on synthetic data provides preliminary empirical support for the theoretical properties of α-RC and points the way toward future extensions to more complex models and real tasks.

One-line Summary

Proposes the α-RC metric, solving PR's scale-dependence problem and achieving an architecture-independent measure of MLP concentration.

Paragraph Summary

This paper addresses the scale-dependence flaw of the Participation Ratio (PR) metric in the Kolmogorov-Arnold Geometry framework by proposing a class of α-RC concentration metrics based on Rényi divergence. PR's maximum value varies with network scale, making it impossible to compare concentration across architectures. α-RC compares normalized minor magnitudes with a uniform distribution via Rényi divergence, normalizing the metric to the [0,1] interval and making it independent of network architecture. This metric family is bounded, monotonic non-decreasing with α, and reduces to an explicit transformation of PR when α=2. Experiments on single-hidden-layer MLPs with synthetic datasets validate the effectiveness of the new metric, providing a more reliable measurement tool for MLP interpretability analysis.