calibration
A metric measuring the agreement between a model's predicted probabilities and observed frequencies. If a model assigns a 70% risk to a group of patients, approximately 70% of them should actually experience the outcome for the model to be well calibrated.
Core Facts
Model calibration is defined in statistics as: if a model claims a certain answer has 70% confidence, then the actual proportion of correct answers among a large number of such claims should be close to 70%.
There is a positive correlation between a model's calibration capability and model scale; larger models can more finely distinguish between states of knowledge.
Timeline (last 90 days)
Calibration is a key metric for evaluating the quality of probabilistic forecasts, referring to the degree of match between predicted probabilities and actual frequencies of occurrence.
The calibration of market prices is usually repeatedly corrected by a large number of traders, whereas the probability outputs of models are prone to overconfidence or conservatism.
Most general-purpose large models suffer from overconfidence issues, where their output probabilities reflect token distribution preferences more than genuine uncertainty estimates.
The calibration of safety guardrails refers to whether the threshold at which a model refuses requests is set appropriately—being overly conservative leads to false rejections of legitimate requests, while insufficient calibration allows harmful content to pass through.
Model calibration is defined in statistics as: if a model claims a certain answer has 70% confidence, then the actual proportion of correct answers among a large number of such claims should be close to 70%.
研究表明校准能力与模型规模之间存在正相关,这一关系在多项实证研究中得到验证
There is a positive correlation between a model's calibration capability and model scale; larger models can more finely distinguish between states of knowledge.
All Facts (7)
Model calibration is defined in statistics as: if a model claims a certain answer has 70% confidence, then the actual proportion of correct answers among a large number of such claims should be close to 70%.
85%VerifiedThere is a positive correlation between a model's calibration capability and model scale; larger models can more finely distinguish between states of knowledge.
75%UnverifiedThe calibration of market prices is usually repeatedly corrected by a large number of traders, whereas the probability outputs of models are prone to overconfidence or conservatism.
50%UnverifiedCalibration is a key metric for evaluating the quality of probabilistic forecasts, referring to the degree of match between predicted probabilities and actual frequencies of occurrence.
50%UnverifiedMost general-purpose large models suffer from overconfidence issues, where their output probabilities reflect token distribution preferences more than genuine uncertainty estimates.
50%UnverifiedThe calibration of safety guardrails refers to whether the threshold at which a model refuses requests is set appropriately—being overly conservative leads to false rejections of legitimate requests, while insufficient calibration allows harmful content to pass through.
50%Unverified研究表明校准能力与模型规模之间存在正相关,这一关系在多项实证研究中得到验证
50%