Mutual Information

Mutual information measures the statistical dependence between two random variables, can capture dependencies of any form, including nonlinear and non-monotonic relationships, and is applicable to combinations of categorical and numerical variables.

In this dataset, after switching to mutual information scores, the feature with the highest mutual information with the target variable is lead_source—a categorical feature.

Mutual information ranges from 0 (completely independent) to the smaller of the variables' entropies (completely dependent), and is often used as a scoring criterion for filter-style feature selection.

Conditional entropy H(Y|X) represents the remaining uncertainty of Y given X, and the relationship between entropy and mutual information, H(Y) = I(X;Y) + H(Y|X), is the key identity for decomposing transient and durable contributions.

Mutual information I(X;Y) is defined as the KL divergence between the joint distribution and the product of the marginal distributions, equivalent to the reduction in uncertainty about Y from knowing X.

In recurrent network auditing, if H_t is the hidden state at time t and X_t is the current input, then I(H_t; X_t) characterizes the transient component, while I(H_t; θ) or I(H_t; training distribution) characterizes the durable component carried by the weights.

Estimating mutual information and conditional entropy under finite samples is extremely challenging: entropy estimation for high-dimensional continuous variables has systematic bias, and commonly used k-nearest-neighbor estimators (such as the Kraskov estimator) remain unstable when the hidden state dimension is high or serial correlation is strong.

Under the multivariate Gaussian distribution, mutual information is fully determined by the covariance matrix and can be expressed as the ratio of the determinant of the joint covariance matrix to the determinant of the marginal covariance matrices.

MPE is formulated in terms of conditional probabilities and prediction accuracy, complementing classical information-theoretic approaches such as mutual information and excess entropy.

Excess entropy is defined as the mutual information between the past and the future, measuring in bits the amount of information in the past that is useful for predicting the future.

互信息定义为I(X;Y) = H(X) + H(Y) - H(X,Y)，度量两个变量间的总依赖关系（包括线性和非线性），为零当且仅当两变量统计独立

Entropy can fully capture information from all moments of a probability distribution, whereas variance can only capture linear, second-order statistical relationships; mutual information can measure total dependence including nonlinear relationships, making it more general than the Pearson correlation coefficient.