NLL
A metric for the quality of probabilistic predictions in classification tasks, defined as the negative logarithm of the predicted probability assigned to the correct class. It is sensitive to the confidence the model assigns to the correct class.
Timeline (last 90 days)
When fine-tuning Qwen on a single RTX 3090, the number of correct answers on the dev set rose from 7/28 to 13/28, but the average negative log-likelihood (NLL) worsened from 2.215 to 3.916.
Accuracy only checks whether the highest-probability option is correct, while NLL also measures the probability assigned to the correct option, so NLL can worsen while accuracy rises.
All Facts (2)
When fine-tuning Qwen on a single RTX 3090, the number of correct answers on the dev set rose from 7/28 to 13/28, but the average negative log-likelihood (NLL) worsened from 2.215 to 3.916.
50%UnverifiedAccuracy only checks whether the highest-probability option is correct, while NLL also measures the probability assigned to the correct option, so NLL can worsen while accuracy rises.
50%