Ridge Regression

In the case of predicting personality from smartphone behavior, according to a priori estimation by the formula, the expected out-of-sample R² of OLS is insufficient to exceed the sample mean baseline (i.e., it is less than or equal to 0), whereas the expected performance of ridge regression can surpass that baseline.

In high-leverage data, the shrinkage mechanism of ridge regression can partially compensate for the predictive loss caused by imbalance.

The paper's analytical framework is based on leave-one-out cross-validation (LOOCV), and derives the expected out-of-sample performance for OLS and ridge regression separately.

Based on deterministic equivalents, hyperparameters (weights for each source and the ridge regression regularization coefficient) can be optimized, and it can be determined whether a private external dataset is useful without accessing the data.

Ridge regression and Lasso can alleviate the coefficient instability caused by multicollinearity, but at the cost of the coefficients no longer being unbiased estimates.

When the prior is a Gaussian distribution N(0, σ²I), taking the logarithm transforms the prior term into an L2 norm penalty on the parameters, which is equivalent to the regularization term in ridge regression; a Laplace prior corresponds to an L1 norm penalty, i.e., Lasso regression.

The author compared four models—linear regression, ridge regression, random forest, and gradient boosting—and all four exhibited exactly the same bias.

Ridge regression shrinks regression coefficients toward zero but does not set them to zero by adding an L2 regularization term to the loss function, whereas LASSO (L1 regularization) produces sparse solutions.

The single-feature holdout IC benchmarks given in the AQuA paper show that combining all features with ridge regression achieves a maximum of only +0.025, and no single feature can exceed 0.03.