Calibration

A model with slightly lower accuracy but well-calibrated confidence is often more suitable for production deployment than a model with higher accuracy that is blindly overconfident.

Model calibration is defined in statistics as: if a model claims a certain answer has 70% confidence, then the actual proportion of correct answers among a large number of such claims should be close to 70%.

Alignment training optimizes the model's statistical behavior in known scenarios rather than establishing formally verifiable security boundaries, and may fail when faced with rare instruction combinations or role-playing frameworks outside the training distribution.

When AI produces incorrect answers, the problem usually lies in the vectorization method or retrieval parameters rather than the model itself.

大模型普遍存在过度自信的问题，需要通过提示词工程和校准策略来校准置信度

Calibration means that the probabilities output by the model should match reality; neural networks generally have an overconfidence tendency, so model probabilities should not be blindly trusted, and thresholds should be set only after validation with labeled data.

校准只能保证统计意义上的覆盖率，不能保证单次预测的可靠性；若模型未见过某种食物，其区间可能看似合理但实际错误。

校准的不确定性区间意味着：若模型声称 90% 置信区间为某范围，则在大量测试样本中约 90% 的真实值应落入该范围。

Specialized models (classifiers, rerankers, entity models) are usually easier to calibrate than an LLM's self-reported confidence, because their training objective directly optimizes the correspondence between scores and correctness.

No single metric deserves complete trust; accuracy and calibration should be used together, and it should be confirmed that evaluation questions are consistent with deployment scenarios.

Calibration does not equal high accuracy; a system that reports 95% confidence on everything may still be overconfident or blindly optimistic even if its accuracy reaches 95%.

The highest-confidence calibration interval is almost useless as a differentiator; the information lives in the middle and bottom where the numbers actually move.

Calibration refers to a judge who is 70% confident in 100 events having approximately 70 of those events actually occur, in which case the judge is said to be well-calibrated.

The calibration phase of conformal prediction requires a large number of verified labels, and the high cost of label verification constitutes a cost bottleneck in practical applications.

Calibration refers to the consistency between the probability estimates a model produces and the actual frequency of occurrence, and is commonly quantified using the Brier Score and calibration curves.

Traditional safety calibration targets a snapshot of a specific model state; once the model weights change through online learning, the correspondence between the original threshold and the new model becomes invalid

Uncalibrated neural networks typically exhibit overconfidence, which is especially pronounced on out-of-distribution data.

Current mainstream large language models are generally overconfident, partly because the training objective optimizes for the fluency of predicting the next word rather than the actual reliability of the output.

Research directions for improving model calibration include introducing uncertainty tokens through supervised fine-tuning, performing post-calibration on probability distributions using temperature sampling, and training models to proactively output the ability to refuse to answer.

Teaching AI systems to express uncertainty appropriately may be just as important as improving their accuracy.