Conformal Prediction

Conformal prediction is an uncertainty quantification framework that outputs prediction sets or prediction intervals with rigorous statistical coverage guarantees.

The distribution-free nature of conformal prediction requires only the exchangeability assumption, meaning the order of data points does not affect their joint distribution.

Conformal prediction can provide theoretically guaranteed coverage rates under finite samples without relying on the underlying model type (random forests, neural networks, or other black-box models).

Conformal prediction can be applied as a wrapper layer on top of any already-trained model without requiring retraining or modification of the model architecture.

In scenarios involving time series, online learning, or concept drift, exchangeability is violated, and extended methods such as weighted conformal prediction or adaptive conformal inference are needed to restore coverage guarantees.

Conformal prediction relies on two key concepts: nonconformity scores and a calibration set, where the calibration set is data with true labels that is independent of the training set.

The efficiency of a prediction set is typically measured by the average size of the prediction set; the smaller the set, the better, provided the coverage rate is satisfied.

The size of a prediction set is a useful signal: a larger set indicates greater model uncertainty, while a smaller set or one containing a single label indicates high model confidence.