Cross-Entropy Loss

A loss function used to measure the difference between probability distributions, commonly applied in classification and language model training. In language models it computes the difference between the predicted probability distribution over the next token and the true distribution, imposing a larger penalty on confident incorrect predictions.