Optimal Neural Network Pruning via Fisher Information Geodesic Distance: A Geometric Perspective

Rethinking Neural Network Pruning Through a Geometric Lens

Model pruning is one of the key techniques for compressing neural networks and reducing inference costs. The core idea is to remove parameters that have the least impact on model performance. For a long time, the most widely used approach has been Magnitude Pruning — simply zeroing out weights with the smallest absolute values. This method is simple and effective, but lacks rigorous theoretical grounding: why does a small weight mean it can be safely removed?

A newly published arXiv paper (arXiv:2609.16129) offers a more fundamental answer. The researchers reframe the pruning problem from the perspective of differential geometry: setting a parameter to zero is essentially moving the model from its current position to the hypersurface where that parameter vanishes. The most natural way to measure the true change caused by this movement is the geodesic distance determined by the Fisher Information Metric in model space.

The Fisher Information Metric is a Riemannian metric on the statistical manifold, introduced by statistician Ronald Fisher. Intuitively, it measures how much the probability distribution represented by a model changes when its parameters undergo a small perturbation. For neural networks, the $(i,j)$-th entry of the Fisher Information Matrix (FIM) describes the joint sensitivity of parameters $\theta_i$ and $\theta_j$. The key advantage of the Fisher metric is that it is coordinate-free — unlike Euclidean distance, which depends on the specific numerical magnitude of parameters, Fisher distance reflects genuine differences at the level of model behavior (i.e., output distributions). This is precisely why it is more appropriate than simple magnitude comparison as a pruning criterion: a weight with a small absolute value, if located in a region highly sensitive to the output distribution, will incur a large "displacement cost" under the Fisher metric, thereby preventing it from being incorrectly pruned.

Core Idea: Pruning as Displacement in Model Space

The key insight of the paper is to transform the operation of "pruning a parameter" into a precisely measurable geometric problem. When you set a parameter to zero, the model "moves" from its current position to the hyperplane where that parameter equals zero. The minimum length of this displacement — the geodesic distance — directly characterizes the true change in the model and its performance after pruning.

In other words, the ideal pruning strategy should prioritize removing parameters that cause the model to "move the least" under the Fisher metric. This provides a clear optimality criterion for pruning, rather than relying on intuition.

A Hierarchy of Optimality Determined by Approximation Quality

The truly elegant aspect is that computing the exact geodesic distance is computationally expensive. By applying progressively finer approximations to this geodesic distance, the paper constructs a hierarchy of optimality for pruning methods:

• The coarsest approximation: exactly recovers traditional magnitude pruning, theoretically explaining why magnitude pruning has worked well for so long — it is a first-order rough approximation of the geometrically optimal solution.
• Intermediate approximations: incorporate local Fisher information, yielding computationally efficient intermediate solutions with near-optimal performance.
• The most faithful approximation: approaches the true geodesic distance, delivering theoretically optimal pruning results.

The value of this hierarchical structure lies not only in providing new methods, but also in unifying pruning approaches of varying complexity under a single mathematical framework. Engineers can make principled trade-offs between accuracy and efficiency based on their computational budget.

Geodesic Distance is the length of the shortest curve between two points on a Riemannian manifold — a generalization of the concept of "straight-line distance" in Euclidean space to curved spaces. On the statistical manifold of parameter space, the geodesic distance between two sets of parameters is not a simple Euclidean norm, but rather the length of the shortest path through a curved space whose curvature is determined by the Fisher Information Matrix. Computing the exact geodesic distance typically requires solving differential equations and is computationally expensive — which is precisely the motivation for the paper's hierarchical approximation scheme. Different levels of approximation correspond to different computational costs, bridging the gap between theoretical optimality and engineering feasibility.

Experimental Validation: Outperforming Baselines Across Multiple Architectures

The paper validates the method on two representative architectures — fully connected networks and Vision Transformers — using the MNIST and CIFAR-10 datasets. Experiments cover the complete pruning ratio range from 0% to 100%, with results averaged across five random seeds to ensure robustness.

The results show that across all "architecture × dataset" combinations examined, the geometric method outperforms both baselines on accuracy and Matthews Correlation Coefficient (MCC):

1. Magnitude-based pruning;
2. Pruning based solely on local Fisher information.

Noteworthy is the full pruning range evaluation. Many pruning methods perform well only at low pruning ratios and degrade sharply under aggressive compression. Testing across the full 0%–100% range more faithfully reflects a method's reliability in aggressive compression scenarios.

Balancing Computational Efficiency and Performance

Beyond pursuing peak accuracy, the paper also highlights the practical value of intermediate approximation schemes. These methods trade a degree of geometric precision for significantly lower computational overhead while maintaining near-optimal performance. For teams deploying models in resource-constrained environments, such "cost-effective" solutions are often more practically relevant than theoretically optimal ones.

Matthews Correlation Coefficient (MCC) is a classification evaluation metric that comprehensively accounts for true positives, true negatives, false positives, and false negatives. It ranges from $[-1, 1]$, where $+1$ represents perfect prediction, $0$ is equivalent to random guessing, and $-1$ indicates completely inverse prediction. Compared to accuracy, MCC is more robust to class imbalance — in pruning experiments, when a model is heavily compressed, its predictive capability for certain classes may degrade severely, while accuracy can be artificially inflated by the masking effect of majority classes. MCC more faithfully reflects the model's overall discriminative ability across all classes. Using both metrics together provides a more comprehensive assessment of true performance degradation at high pruning ratios.

Significance: Providing Mathematical Justification for Pruning

This work's contributions can be understood on two levels.

At the methodological level, it proposes a pruning approach that achieves state-of-the-art results experimentally and is not limited to a single architecture — it works effectively on both traditional networks and modern Transformers.

At the theoretical level, it provides a mathematically motivated, empirically validated justification for pruning. In the past, pruning was largely an empirically driven exercise. This geometric framework reveals the unifying principle behind different pruning criteria — they are simply the same geodesic distance expressed at different levels of approximation precision. This deeper understanding of "why it works" has significant guiding value for designing better compression algorithms in the future.

Summary

This paper reformulates neural network pruning in the language of differential geometry: pruning is a displacement of the model in Fisher information metric space, and optimal pruning corresponds to the shortest geodesic distance. Through hierarchical approximation, it brings the entire spectrum from magnitude pruning to complex geometric methods under a unified framework. Experiments on fully connected networks and Vision Transformers across MNIST and CIFAR-10 demonstrate comprehensive improvements over magnitude pruning and local Fisher methods. For researchers and engineers focused on model compression and efficient inference, this geometric perspective offers both practical tools and theoretical clarity.