The Information-Geometric Blind Spot of Conformal Test Martingales: Why Distribution Drift Goes Undetected

The Blind Spot Mechanism of Conformal Test Martingales

Conformal Test Martingales (CTMs) are e-processes for sequentially testing exchangeability, testing via p-values produced by a conformal function A. However, any distribution drift that keeps the pushforward distribution P∘A⁻¹ unchanged will leave the p-value distribution unchanged, and is therefore invisible to all CTMs based on A. In an information-geometric framework, this blind spot is formalized as the A-equivalence class [P]_A, revealing a deep connection to its nearest element under KL divergence—the A-secret I-projection.

The core of a CTM lies in using the conformal function A to map high-dimensional data into one-dimensional p-values, which are then used to construct an e-process. But this dimensionality-reduction process inherently loses information: as long as the drift does not change the marginal distribution of the conformal scores, the statistical behavior of the p-value sequence remains completely unchanged. In other words, a CTM can only detect drifts that change the pushforward distribution of A, and is powerless against drifts that are "invisible" under A. This blind spot is not due to insufficient statistical power but is a fundamental limitation determined by the definition of the conformal function.

e-processes and conformal p-values: An e-process is a class of nonnegative martingales (or supermartingales) whose expectation under the null hypothesis does not exceed 1, thereby providing an anytime-valid sequential test via Ville's inequality. A Conformal Test Martingale (CTM) is a special case of an e-process: it uses a conformal function A to map each data point to a conformal score, then computes conformal p-values and combines them into an e-process. The construction of conformal p-values guarantees that, under the exchangeability null hypothesis, the p-value sequence is valid (i.e., super-uniform), so the CTM can control the type I error at any stopping time. Understanding this is a prerequisite for grasping the blind spot mechanism: the validity of a CTM depends only on the distributional properties of the conformal p-values, and that distribution is fully determined by the pushforward distribution of the conformal scores under the null hypothesis. Hence any drift that does not change this pushforward distribution will not affect the statistical behavior of the p-value sequence.

A-Equivalence Classes and Secret I-Projections

The paper formalizes the blind spot via the A-equivalence class [P]_A: all distributions that share the same pushforward distribution as P under A form an equivalence class. For any target distribution T, this equivalence class contains a unique KL-nearest element Q*, called the A-secret I-projection, whose density has the explicit expression q*(z) = t(z)p_S(A(z))/t_S(A(z)). This result decomposes distribution drift within the blind spot into a detectable component and a secret component, and yields a decomposition analogous to the Pythagorean theorem. When A is taken to be the Neyman-Pearson likelihood ratio, the secret component vanishes, consistent with known results on optimal e-value design, indicating that the size of the blind spot depends on the choice of conformal function.

I-projections and KL divergence in information geometry: Information geometry treats families of probability distributions as statistical manifolds, with KL divergence acting as an asymmetric "distance." Given a distribution T and a constraint set (here the A-equivalence class [P]_A), the I-projection refers to finding the distribution Q* in the constraint set that minimizes the KL divergence D(T||Q). Since KL divergence is not symmetric, the I-projection generally differs from the reverse I-projection (which minimizes D(Q||T)). The A-secret I-projection in the paper is precisely the result of projecting T onto [P]_A, and its density expression q*(z) = t(z)p_S(A(z))/t_S(A(z)) can be derived via variational methods or by introducing Lagrange multipliers for the constraint. What makes this projection "secret" is that Q* and P share the same pushforward distribution under A, so a CTM cannot distinguish between them, yet the support of Q* in the original data space may be entirely different from that of P. The Pythagorean-like decomposition shows that the KL divergence between T and P can be decomposed into the divergence from T to Q* plus the divergence from Q* to P, where the first term corresponds to the detectable component and the second to the secret component.

Neyman-Pearson likelihood ratio and optimal e-values: In simple hypothesis testing, the Neyman-Pearson lemma states that the likelihood ratio is the optimal test statistic. In the e-value framework, the likelihood ratio is itself an e-value and is optimal in certain senses (e.g., it has the largest expected log growth under the alternative hypothesis). When the conformal function A is taken to be the likelihood ratio, the A-equivalence class degenerates to a singleton (or an extremely restricted set), the secret component vanishes, and the blind spot of the CTM is minimized. This connection reveals a deep trade-off between the choice of conformal function and test power: the closer A is to the likelihood ratio, the smaller the blind spot—but the likelihood ratio typically depends on the unknown alternative distribution and is hard to obtain exactly in practice, so a compromise must be made between blind spot size and computability.

Empirical Evidence on High-Dimensional Image Data

To validate the theory, the authors empirically construct this I-projection on high-dimensional image data. The experiments use the EMNIST letters and MNIST digits datasets: by conditionally resampling EMNIST letters so that their conformal scores exactly match those of MNIST digits, they construct a distribution drift whose support is disjoint from the original distribution yet which renders the CTM mathematically completely blind. This experiment intuitively demonstrates the reality of the blind spot—even if the data distribution changes significantly, as long as the conformal scores are aligned, the CTM's test statistic shows no reaction whatsoever.

Implications for e-Value Design and Practice

This research reveals the quantitative relationship between the CTM blind spot and the choice of conformal function. When the Neyman-Pearson likelihood ratio is used as the conformal function, the blind spot vanishes, providing theoretical guidance for designing more robust e-values: in practice, one should choose conformal functions that retain as much discriminative information as possible, or combine multiple conformal functions to build a composite test. Meanwhile, the explicit construction of the A-secret I-projection also provides a computable tool for evaluating the blind spot size of existing CTMs, helping to quantify risk in distribution drift detection scenarios.

Limitations and Outlook

The current work focuses on blind spot analysis for exchangeability testing; for more general sequential testing scenarios, the characterization of the blind spot may be more complex. Moreover, the trade-off between the computational cost of conformal functions in high-dimensional settings and the blind spot size still requires further study. Future directions include developing adaptive conformal function selection strategies and extending information-geometric tools to other e-process constructions.

One-line Summary

Conformal test martingales have a blind spot due to dimensionality reduction by the conformal function, characterized information-geometrically as A-equivalence classes and KL-nearest projections.

Paragraph Summary

This article explores the blind spot of Conformal Test Martingales (CTMs) in sequentially testing exchangeability. A CTM maps data to p-values via a conformal function A and constructs an e-process, but any distribution drift that keeps the pushforward distribution P∘A⁻¹ unchanged will leave the p-value distribution unchanged and thus be invisible to the CTM. The authors formalize this blind spot in an information-geometric framework as the A-equivalence class [P]_A, and prove that this equivalence class contains a unique KL-nearest element—the A-secret I-projection—whose density has an explicit expression. This projection decomposes distribution drift into a detectable component and a secret component, and gives a Pythagorean-like decomposition. When A is the Neyman-Pearson likelihood ratio, the secret component vanishes, consistent with known results on optimal e-value design. Experiments on high-dimensional image data (EMNIST/MNIST) intuitively demonstrate the reality of the blind spot: by conditionally resampling to align conformal scores, one can construct a distribution drift with disjoint support that renders the CTM completely blind. The research provides theoretical guidance for robust e-value design and points out that the blind spot size depends on the choice of conformal function.