ROC-AUC
A standard evaluation metric that measures the overall discriminative capability of a binary classification model across different decision thresholds. Values range from 0 to 1, where 0.5 indicates random guessing and 1.0 represents perfect classification. It is widely used for assessing machine learning model performance.
Timeline (last 90 days)
ReachNT's urgency model achieves a ROC-AUC of 0.84 on unseen phrasing, and as part of an overall system that includes human review it catches 99.3% of dangerous work orders.
Calibration and discrimination are two independent metrics: discrimination measures the model's ability to rank positives ahead of negatives (such as ROC-AUC), while calibration measures whether the model's output probabilities match true frequencies; commonly used calibration measures include Expected Calibration Error (ECE) and reliability diagrams, and correction methods include Platt scaling, isotonic regression and temperature scaling.
A learner, when detecting forged receipts on this dataset, found that regardless of how the model was adjusted, the ROC-AUC consistently hovered between 0.46 and 0.54, close to random guessing.
The approach using frozen visual features yielded a ROC-AUC of only 0.482, meaning the model essentially learned nothing.
All Facts (4)
ReachNT's urgency model achieves a ROC-AUC of 0.84 on unseen phrasing, and as part of an overall system that includes human review it catches 99.3% of dangerous work orders.
50%UnverifiedCalibration and discrimination are two independent metrics: discrimination measures the model's ability to rank positives ahead of negatives (such as ROC-AUC), while calibration measures whether the model's output probabilities match true frequencies; commonly used calibration measures include Expected Calibration Error (ECE) and reliability diagrams, and correction methods include Platt scaling, isotonic regression and temperature scaling.
50%UnverifiedThe approach using frozen visual features yielded a ROC-AUC of only 0.482, meaning the model essentially learned nothing.
50%UnverifiedA learner, when detecting forged receipts on this dataset, found that regardless of how the model was adjusted, the ROC-AUC consistently hovered between 0.46 and 0.54, close to random guessing.
50%