Conformal Risk Control
Conformal risk control is an extension of conformal prediction that lets users define a general loss function, such as expected false alarm rate, and adjusts prediction thresholds using empirical risk on a calibration set, providing a statistical guarantee under finite samples that expected loss does not exceed a target level. The guarantee typically relies on an exchangeability assumption, and effective coverage changes when the model is allowed to abstain.
Timeline (last 90 days)
The study employs conversation-level conformal risk control to constrain the expected loss of commitment predictions, and this guarantee was met in all twelve configurations tested.
The guarantee of conformal risk control is bought through abstention; on the most difficult corpus, coverage and alert recall dropped sharply.
Conformal risk control is an extended framework of conformal prediction that allows users to define more general loss functions and provides statistical guarantees under finite samples, but its guarantees usually rely on the exchangeability assumption.
Meeting risk control metrics does not equal system usability; when the abstention rate is too high, the practical value of predictions is greatly diminished.
All Facts (4)
The study employs conversation-level conformal risk control to constrain the expected loss of commitment predictions, and this guarantee was met in all twelve configurations tested.
50%UnverifiedThe guarantee of conformal risk control is bought through abstention; on the most difficult corpus, coverage and alert recall dropped sharply.
50%UnverifiedConformal risk control is an extended framework of conformal prediction that allows users to define more general loss functions and provides statistical guarantees under finite samples, but its guarantees usually rely on the exchangeability assumption.
50%UnverifiedMeeting risk control metrics does not equal system usability; when the abstention rate is too high, the practical value of predictions is greatly diminished.
50%