Entropic Scree: Reconstructing PCA Dimensionality Reduction by Replacing Variance with Information Entropy

When PCA Meets Messy Real-World Data

Principal Component Analysis (PCA) is one of the most classic dimensionality reduction methods in the data science toolkit. It projects high-dimensional data into a lower-dimensional space by finding the directions of maximum variance, reducing data complexity while preserving key information. However, PCA has a deeply ingrained assumption — its way of measuring information is built on linear variance.

The mathematical essence of PCA is eigenvalue decomposition of the data covariance matrix (or equivalently, Singular Value Decomposition, SVD, of the data matrix). The covariance matrix only encodes second-order statistical relationships between variables (i.e., linear correlations), which means PCA can inherently only discover linear manifold structures. When data lies on a curved nonlinear manifold — such as the famous Swiss Roll dataset — PCA projections severely distort the structure, because it tries to fit a curved surface with a hyperplane. Additionally, PCA is extremely sensitive to outliers, since variance itself is easily inflated by extreme values.

This assumption works well on textbook-clean datasets, but real-world data often refuses to cooperate: mixed data types (continuous and categorical values interleaved), highly nonlinear generative processes, extremely low signal-to-noise ratios, and so-called "sparsity" problems (far more variables than samples). In these scenarios, traditional PCA's performance degrades noticeably, because linear variance cannot faithfully reflect the information structure embedded in the data.

Recently, a researcher open-sourced a new method called Entropic Scree, attempting to revisit this classic dimensionality reduction problem from an information-theoretic perspective.

From Variance to Information Entropy: The Core Idea Behind Entropic Scree

Replacing Linear Variance with Information Theory

The core innovation of Entropic Scree is: using information theory instead of linear variance as the measurement foundation — think of it as "an information-theoretic upgrade to PCA."

The traditional PCA "Scree Plot" displays the proportion of variance explained by each principal component, and researchers use the "elbow method" to determine how many dimensions to retain. The scree plot was introduced by psychometrician Raymond Cattell in 1966, originally used to determine the number of factors in factor analysis. It plots the eigenvalues corresponding to each principal component in descending order as a line chart, with its name derived from the geological formation of rock debris scattered at the base of a mountain — a steep slope with gentle scree below. The Elbow Method is the most common way to interpret a scree plot: find the inflection point where the curve transitions from steep to flat, and principal components beyond that point are considered to primarily reflect noise rather than signal. However, the elbow method is inherently a subjective visual judgment lacking rigorous statistical criteria, and is especially difficult to apply when eigenvalue decay is smooth, making it hard to identify a clear cutoff point.

Entropic Scree, as its name suggests, builds this analysis on Entropy — by measuring the actual information content carried by each dimension, rather than mere variance spread, to estimate the data's "intrinsic rank." Intrinsic dimensionality refers to the minimum number of degrees of freedom needed to describe the data, which is typically much lower than the data's apparent dimensionality. Intrinsic dimension estimation is one of the core problems in manifold learning, and multiple approaches have been proposed: neighbor-distance-based methods (such as Maximum Likelihood Estimation of Intrinsic Dimension), topology-based methods (such as persistent homology), and projection-based methods. Entropic Scree attempts to provide a more robust intrinsic dimension estimate through an information-theoretic framework, transforming the judgment criterion from a subjective visual "elbow" to a quantitative metric grounded in information theory.

The Advantages of Information Entropy as a Measure

Information entropy is a more universal measure of "information content." Information entropy was introduced by Claude Shannon in 1948 in his foundational paper A Mathematical Theory of Communication. For a discrete random variable X, Shannon entropy is defined as H(X) = -Σ p(x) log p(x), which measures the average amount of information needed to describe the outcome of a random variable (in bits or nats). Intuitively, higher entropy means greater uncertainty and richer information content. For continuous variables, the corresponding concept is differential entropy.

Unlike variance, entropy can fully capture information from all moments of a probability distribution, not just the mean and variance. Variance can only capture linear, second-order statistical relationships, while entropy-based measures can theoretically capture more complex nonlinear dependencies. Mutual Information, a concept derived from entropy, measures the total dependency between two variables (including both linear and nonlinear), making it a more general dependency measure than the Pearson correlation coefficient — when a complex non-monotonic relationship exists between two variables, the correlation coefficient may be close to zero, but mutual information correctly captures this dependency.

This means that when the true structure of data is hidden in nonlinear relationships, Entropic Scree has the potential to reveal information that traditional PCA would miss.

According to the author, this method demonstrates stronger robustness for the following types of challenging data:

• Mixed data types: No need to forcibly convert categorical variables to numerical values (traditional PCA requires all inputs to be continuous numerical values; categorical variables typically need one-hot encoding or other preprocessing, which artificially introduces spurious distance relationships)
• Highly nonlinear generative processes: Not constrained by linear assumptions
• Low signal-to-noise ratio scenarios: More robustly extracts signals from noise
• Sparse high-dimensional data: Works effectively even when variables outnumber samples

The author also emphasizes that these advantages "compound as scale and system complexity increase" — the larger and more complex the data, the more pronounced Entropic Scree's advantages over traditional PCA may become.

Practical Application: Precisely Sizing Neural Network Bottleneck Layers

One of the most engineering-valuable applications of this method is precisely determining the dimensionality of neural network bottleneck layers.

Take autoencoders as an example: the dimensionality of the middle bottleneck layer determines how much the model compresses the data. An autoencoder is a neural network that learns compressed representations of data through an encoder-decoder architecture: the encoder maps the input to a low-dimensional bottleneck layer (also called the latent space), and the decoder reconstructs the input from the bottleneck representation. Choosing the bottleneck dimensionality is fundamentally a tradeoff between information compression rate and reconstruction quality, corresponding to Rate-Distortion Theory in information theory — Shannon proved in 1959 that for a given distortion tolerance, there exists a theoretical lower bound on the minimum coding rate.

Set this dimension too large, and compression is insufficient — the model may simply memorize the data (the latent space may develop "dead zones" where some dimensions are not effectively utilized). Set it too small, and critical information is lost, leading to blurry reconstructions. In Variational Autoencoders (VAE), KL divergence regularization partially alleviates this issue, but the bottleneck dimension still needs to be manually specified. Traditional approaches often rely on experience or grid search to iteratively probe this hyperparameter, which is computationally expensive and lacks theoretical guidance.

If Entropic Scree can first estimate the precise intrinsic dimensionality of a dataset, engineers can explicitly determine the bottleneck layer size based on this, giving architecture design a clear information-theoretic basis rather than relying on trial and error. This is especially valuable in practice for representation learning and feature compression on tabular data — tabular data typically contains mixed-type features, and its intrinsic manifold structure lacks the mature prior knowledge available for image data.

Open-Source Implementation and Reproducibility

The author has fully open-sourced the work, demonstrating commendable research transparency:

• Code repository: Implemented in R, hosted on GitHub (tjleestjohn/Entropic-Scree)
• Preprint paper: Published on Zenodo (DOI: 10.5281/zenodo.22028087)

The author actively invites community feedback and independent verification — this open stance is a necessary step for any new method on its path to maturity.

Limitations That Deserve a Sober Assessment

As a new method published in preprint form, Entropic Scree has not yet undergone large-scale peer review or community-wide practical validation. The many advantages mentioned still need to be independently verified across diverse public datasets and benchmark tests.

Furthermore, information-theoretic methods typically involve estimating probability densities or mutual information, which is itself a technical challenge in high-dimensional, finite-sample settings — the so-called "curse of dimensionality." Kernel Density Estimation (KDE) degrades sharply when dimensionality exceeds 5-10, because the required sample size grows exponentially with dimension. In recent years, researchers have developed multiple alternatives: k-nearest-neighbor-based mutual information estimators (such as the Kraskov-Stögbauer-Grassberger estimator, known as the KSG estimator), random-forest-based conditional mutual information estimation, and neural-network-based MINE (Mutual Information Neural Estimation) methods. Each method makes different tradeoffs in bias-variance balance, computational complexity, and applicable scenarios. The specific density/entropy estimation strategy adopted by Entropic Scree will directly impact its practicality and computational efficiency on large-scale high-dimensional data.

Computational cost and estimation stability are concerns for real-world deployment. While the R language implementation is convenient for the statistics community, production-scale environments may require high-performance ports in Python or C++.

Conclusion: A Fresh Perspective on Dimensionality Reduction Through Information Theory

Entropic Scree represents a valuable research direction: replacing overly simplified linear assumptions in classic algorithms with more universal information measures. For practitioners who have long struggled with mixed-type, nonlinear, high-noise tabular data, it offers a new tool worth trying — especially when precise estimation of data's intrinsic dimensionality is needed to guide neural network architecture design.

From a broader perspective, the information-theoretic approach represented by Entropic Scree resonates with other recent trends in machine learning: the Information Bottleneck theory's interpretation of deep learning, the Minimum Description Length (MDL) principle's application in model selection, and Information Geometry's use in optimization algorithms all point to information theory becoming an important bridge connecting statistical learning and algorithm design.

Whether the method can withstand the test of widespread practice remains to be seen, but the fully open-sourced code and paper provide a low-barrier entry point for researchers and engineers to verify it independently — try it on your own data and see whether entropy-based dimensionality reduction reveals the true structure of your data better than traditional PCA.