ECE

Expected Calibration Error (ECE) bins prediction samples by confidence level, computes the weighted average of the difference between predicted confidence and actual accuracy within each bin, and ideally approaches 0.

Model calibration refers to the degree to which a model's output probabilities match its actual accuracy rate; for a well-calibrated model, if the probability it gives for a certain type of question is 0.8, then about 80% of such questions are indeed answered correctly.

On calibration, Jev's ECE in independent tests varies between 0.02 and 0.35 depending on the task; Laya's ECE is 0.466 before temperature fitting (overconfident) and drops to 0.081 after fitting; Drex has not published calibration data.

On 300 practice exam papers labeled by Qwen3-32B, fine-tuned v3 of Laya raised the judgment macro-F1 across 11 questions from 0.33 to 0.57 versus the zero-shot version, cut calibration error from 0.15 to 0.07, and reduced danger-level MAE from 2.78 to 1.56.

Calibration and discrimination are two independent metrics: discrimination measures the model's ability to rank positives ahead of negatives (such as ROC-AUC), while calibration measures whether the model's output probabilities match true frequencies; commonly used calibration measures include Expected Calibration Error (ECE) and reliability diagrams, and correction methods include Platt scaling, isotonic regression and temperature scaling.

The confidence column shipped by Largha-150M-v1 has an ECE of 0.2519, and the built-in recalibration column (temperature scaling) reduces it to 0.0168.

The escalation gate based on the confidence threshold τ must first be recalibrated with temperature scaling, otherwise the threshold is invalid, because the shipped confidence column has an ECE of 0.2519.

ECE (Expected Calibration Error) measures the consistency between a model's confidence and its true accuracy; a lower ECE indicates more trustworthy probability outputs.

The Expected Calibration Error (ECE) increased in seven out of eight event pairs, and in affected scenarios, 12%–51% of the area was confidently mislabeled by the model with near-zero accuracy.

Point-estimate confidence fails under distribution shift, whereas Bayesian uncertainty measures such as MC Dropout can genuinely identify signal failure.

On a test set of 333 articles and 1,218 prompts, Fiorillo v0.5 achieved an Expected Calibration Error (ECE) of 0.0168, well below the preset upper limit of 0.05.

On top of the Brier score, RLCD adds a per-bin calibration penalty term similar to the ECE approach, reducing rewards in overconfident intervals and increasing rewards in underconfident intervals.

There is an unverified risk with calibration rewards: they may simply flatten the confidence of all outputs toward a middle value, performing well on the ECE metric while losing discriminative information.

The author found that using only a global -λ·ECE penalty across the batch is nearly ineffective, because it is directly absorbed and offset by the PPO or GRPO baseline.

Confidence calibration is typically measured using reliability diagrams and Expected Calibration Error (ECE).

A well-calibrated model is one where the actual accuracy of predictions with output probability P is also close to P across all such predictions; calibration quality is commonly measured using the ECE (Expected Calibration Error) metric.

校准是指模型给出的置信度与实际频率相符的程度，常通过可靠性图和期望校准误差（ECE）来量化