Closed-Form Solution for Gaussian BROJA Information Decomposition: 100x Speedup for High-Dimensional PID

The Core Challenge in Information Decomposition

Quantifying the individual contributions of two information sources that jointly carry information about a target variable has long been a fundamental challenge in information theory. BROJA Partial Information Decomposition (PID) provides a framework with clear operational semantics, decomposing the information that two sources carry about a target into three components: unique information, redundant information, and synergistic information.

This decomposition has broad applications in neuroscience, machine learning, and complex systems analysis. It can answer foundational questions such as "how much information does each brain region contribute independently?" or "do two features exhibit synergistic effects?"

Yet theoretical elegance does not guarantee computational feasibility. For continuous variables — especially high-dimensional ones — computing the BROJA decomposition requires solving a highly challenging optimization problem. This computational bottleneck has long prevented the method from being applied to large-scale, real-world data.

PID was introduced by Williams and Beer in 2010, motivated by a key limitation of Shannon mutual information: it can only tell us "how much information do two sources jointly carry about the target," but cannot distinguish whether that information is redundant (both sources can independently predict the target) or complementary (only their combination enables prediction). The BROJA formulation (Bertschinger, Rauh, Olbrich, Jost, Ay) is one of the most widely accepted PID implementations. Its defining feature is that redundant information is defined via a constrained optimization problem: among all joint distributions satisfying certain marginal constraints, find the one that maximizes redundancy. This definition carries a clear operational meaning — redundancy corresponds to "information that cannot be eliminated regardless of which source is observed." The three components satisfy an additive constraint: the total mutual information from both sources equals the sum of unique information, redundant information, and synergistic information. Notably, synergistic information can be negative (when the two sources interfere with each other), which importantly departs from the naive intuition that "information is additive."

From Numerical Optimization to Closed-Form Solution

Prior work had already made partial progress: when each pair of source–target variables jointly follows a Gaussian distribution, the intractable optimization problem can be reduced to a finite-dimensional numerical optimization. This result made BROJA decomposition computationally feasible in the Gaussian case, but still required an iterative numerical solving process.

The key contribution of this work goes one step further — the researchers directly derive a closed-form solution to the optimization problem. In other words, rather than iteratively approximating the optimal solution, the result can be computed directly through an analytic expression.

What a Closed-Form Solution Really Means

The value of a closed-form solution lies not only in mathematical completeness, but in the practical computational efficiency it enables. Numerical optimization comes with a host of uncertainties — convergence behavior, sensitivity to initialization, number of iterations — whereas an analytic expression is deterministic and computed in a single step. For research scenarios requiring repeated PID computation (e.g., across large-scale neural recordings or high-dimensional feature spaces), this difference is dramatically amplified.

In mathematics and computation, a "closed-form solution" refers to an analytic expression written using a finite number of elementary operations (addition, multiplication, exponentiation, logarithms, matrix operations, etc.), as opposed to a numerical solution obtained through iterative algorithms (e.g., gradient descent, Newton's method). For the Gaussian BROJA problem, prior numerical methods relied on algorithms such as the Alternating Direction Method of Multipliers (ADMM) or similar convex optimization routines, requiring dozens to hundreds of iterations per computation, with accuracy controlled by convergence thresholds. The closed-form solution compresses this process into a fixed number of matrix operations: given the covariance matrix, substitute it into the formula and the result is immediate. Computational complexity drops from (number of iterations × per-step cost) to the cost of a single matrix decomposition. This advantage is especially pronounced in high dimensions, where the covariance matrix grows quadratically with the number of variables, while linear algebra operations like eigendecomposition are supported by highly optimized numerical libraries.

Two Orders of Magnitude in Performance Gains

The closed-form expression applies to general multivariate Gaussian settings, enabling over two orders of magnitude (100x or more) of speedup in high-dimensional Gaussian BROJA PID computation.

This level of acceleration is practically decisive. Computations that might previously have taken hours or longer can potentially be completed in seconds or minutes. For researchers who rely on information decomposition as an analytical tool, this means handling larger datasets, conducting more intensive parameter sweeps, or embedding PID into more complex analysis pipelines without worrying about computation time becoming a bottleneck.

Scope and Prerequisites

It is important to note that this closed-form solution assumes a multivariate Gaussian setting. The Gaussian assumption is a reasonable approximation in many practical scenarios and is a common simplification in information-theoretic analysis. However, for strongly non-Gaussian, heavy-tailed, or highly nonlinear data, the direct applicability of this method warrants careful consideration. This also opens avenues for future research — extending the closed-form approach to more general distribution families.

Open-Source Implementation and Reproducibility

The researchers have released the associated code as an open-source project named Eigen_PID, hosted on GitHub at https://github.com/brainsandmachines/Eigen_PID. Open-sourcing the implementation not only enhances the reproducibility of the research, but also lowers the barrier for other researchers to adopt the method.

The project name suggests that the derivation of the closed-form solution is likely closely tied to the eigenvalue/eigenvector structure (eigen-structure) of the covariance matrix — a natural mathematical tool in multivariate Gaussian information calculations. Interested researchers can use the repository to directly verify the paper's results and apply them to their own data.

The "Eigen" in "Eigen_PID" is not merely a reference to the C++ linear algebra library Eigen — it more directly hints at the mathematical structure of the closed-form solution. Mutual information under a multivariate Gaussian distribution is entirely determined by the covariance matrix, and the information content of a covariance matrix is fundamentally encoded in its eigenvalues. Specifically, the mutual information between Gaussian variables X and Y can be expressed as a ratio of determinants of the joint and marginal covariance matrices, and a determinant equals the product of all eigenvalues. The constraints in the BROJA optimization problem (fixing marginal distributions) translate, in the Gaussian case, into constraints on specific submatrices — making it possible to find the analytic optimum in the eigenspace. This mathematical pathway shares algebraic structure with classical results such as Gaussian channel capacity and Principal Component Analysis (PCA), representing a deep intersection of information geometry and linear algebra.

Implications for Research Practice

At its core, this work is a contribution toward "making a theoretical tool genuinely usable." The conceptual framework of BROJA PID has long existed and its operational meaning is well-recognized, but computational cost has been a real barrier to its widespread adoption. By reducing the core optimization problem to a closed-form solution, this research transforms a method that was feasible in principle but expensive in practice into one that can be used routinely.

For information flow analysis in neuroscience, feature interaction diagnosis in machine learning, and any scenario requiring quantification of information redundancy and synergy within a high-dimensional Gaussian framework, this is a highly practical advance. The efficiency gain is not merely about "computing faster" — it may fundamentally change the scale of problems and the combination of methods that researchers are willing to explore.