Conformal test martingale
An e-process constructed from conformal p-values, used for sequential testing of data exchangeability, with anytime-valid type I error control.
Timeline (last 90 days)
The conformal test martingale (CTM) is a special case of the e-process, in which a conformal function A maps each data point to a conformal score and computes a conformal p-value, which are then combined into an e-process for sequential testing.
Any distribution shift that keeps the pushforward distribution P∘A⁻¹ unchanged leaves the distribution of the conformal p-values unchanged, and is thus invisible to all CTMs based on the conformal function A.
This blind spot is not due to insufficient statistical power but is a fundamental limitation determined by the definition of the conformal function.
The construction of conformal p-values guarantees that under the exchangeability null hypothesis the sequence of p-values is super-uniform, so the CTM can control the type I error at any stopping time.
When the conformal function A is the Neyman-Pearson likelihood ratio, the secret component vanishes, the A-equivalence class degenerates to a single point or an extremely restricted set, and the CTM's blind spot is minimized.
The experiment uses the EMNIST letters and MNIST digits datasets, constructing a distribution shift that is disjoint from the support of the original distribution yet completely blinds the CTM by conditionally resampling the EMNIST letters so that their conformal scores exactly match those of the MNIST digits.
The explicit construction of the A-secret I-projection provides a computable tool for assessing the size of the blind spot of existing CTMs, helping to quantify risk in distribution shift detection scenarios.
All Facts (7)
The construction of conformal p-values guarantees that under the exchangeability null hypothesis the sequence of p-values is super-uniform, so the CTM can control the type I error at any stopping time.
50%UnverifiedAny distribution shift that keeps the pushforward distribution P∘A⁻¹ unchanged leaves the distribution of the conformal p-values unchanged, and is thus invisible to all CTMs based on the conformal function A.
50%UnverifiedThis blind spot is not due to insufficient statistical power but is a fundamental limitation determined by the definition of the conformal function.
50%UnverifiedThe conformal test martingale (CTM) is a special case of the e-process, in which a conformal function A maps each data point to a conformal score and computes a conformal p-value, which are then combined into an e-process for sequential testing.
50%UnverifiedWhen the conformal function A is the Neyman-Pearson likelihood ratio, the secret component vanishes, the A-equivalence class degenerates to a single point or an extremely restricted set, and the CTM's blind spot is minimized.
50%UnverifiedThe explicit construction of the A-secret I-projection provides a computable tool for assessing the size of the blind spot of existing CTMs, helping to quantify risk in distribution shift detection scenarios.
50%UnverifiedThe experiment uses the EMNIST letters and MNIST digits datasets, constructing a distribution shift that is disjoint from the support of the original distribution yet completely blinds the CTM by conditionally resampling the EMNIST letters so that their conformal scores exactly match those of the MNIST digits.
50%