Learn Then Test
A threshold selection framework based on multiple hypothesis testing proposed by Angelopoulos et al. It treats all candidate thresholds as hypotheses to be tested, computes risk upper bounds using finite-sample p-value constructions, and selects the set of thresholds satisfying the risk budget through an FDR-controlling procedure.
Timeline (last 90 days)
In settings where the test risk is definable, the naive threshold-selection baseline exceeds the target risk level in about half of the cases, whereas the Learn Then Test (LTT) method exceeds it in only 7.8% of the cases.
Learn Then Test (LTT) is a threshold-selection framework based on multiple hypothesis testing proposed by Angelopoulos et al., whose core idea is to treat all candidate thresholds as a set of hypotheses to be tested, use finite-sample p-value construction methods to compute the risk upper bound corresponding to each threshold, and then use an FDR control procedure to select in one pass the set of thresholds that satisfy the risk budget.
LTT does not rely on the premise that the probabilistic scores are exactly accurate, but instead provides a finite-sample risk guarantee: as long as the set of candidate thresholds is fixed in advance, the probability that the threshold selected by LTT exceeds the target risk on the true distribution is strictly controlled.
The implications for LLM engineering practice include: do not directly treat the probabilities output by the model as decision confidence; in scenarios where handoff to a human is not possible, a conservative fallback strategy should be explicitly designed; threshold selection requires statistical guarantees.
All Facts (4)
In settings where the test risk is definable, the naive threshold-selection baseline exceeds the target risk level in about half of the cases, whereas the Learn Then Test (LTT) method exceeds it in only 7.8% of the cases.
50%UnverifiedLearn Then Test (LTT) is a threshold-selection framework based on multiple hypothesis testing proposed by Angelopoulos et al., whose core idea is to treat all candidate thresholds as a set of hypotheses to be tested, use finite-sample p-value construction methods to compute the risk upper bound corresponding to each threshold, and then use an FDR control procedure to select in one pass the set of thresholds that satisfy the risk budget.
50%UnverifiedLTT does not rely on the premise that the probabilistic scores are exactly accurate, but instead provides a finite-sample risk guarantee: as long as the set of candidate thresholds is fixed in advance, the probability that the threshold selected by LTT exceeds the target risk on the true distribution is strictly controlled.
50%UnverifiedThe implications for LLM engineering practice include: do not directly treat the probabilities output by the model as decision confidence; in scenarios where handoff to a human is not possible, a conservative fallback strategy should be explicitly designed; threshold selection requires statistical guarantees.
50%