In PyTorch, both CrossEntropyLoss and NLLLoss (Negative Log Likelihood Loss) are widely used for training classification models. They are commonly used together in a softmax regression setup where the task is to predict a single class label for each input. Despite their similarities, there are significant differences between the two loss functions, which we will discuss below.
**CrossEntropyLoss**
CrossEntropyLoss is commonly used in multi-class classification problems, where the model is required to predict one class label from a set of possible labels. In this loss function, the outputs are assumed to follow a class probability distribution, and the loss is calculated by comparing the predicted class probabilities against the true distribution. This loss function applies a softmax function to the output layer activations and then calculates the cross-entropy between the predicted probability distribution and the true distribution. The cross-entropy measures the difference between two probability distributions and is given by:
$CE(mathbf{y},mathbf{hat{y}})=-sum{mathbf{y}log(mathbf{hat{y}})}$where y is the true distribution and ŷ is the predicted distribution.
Technically, CrossEntropyLoss combines two functions - softmax and negative log-likelihood loss - into a single equation. In PyTorch, we can easily compute this loss by applying nn.CrossEntropyLoss() to the output of the model and the true labels.
**NLLLoss**
Like CrossEntropyLoss, the NLLLoss function is also used in multi-class classification problems. The difference here is that instead of applying a softmax function to the output layer, NLLLoss expects the inputs to be log-probabilities. This means that the output of the model should be the log of the predicted class probabilities. NLLLoss is closely related to CrossEntropyLoss and is computed as follows:
$NLL(mathbf{y},mathbf{hat{y}})=-sum{mathbf{y}log(mathbf{hat{y}})}$where y is the true distribution and ŷ is the predicted log-probability distribution.
In simpler terms, NLLLoss calculates the negative log-likelihood of the predicted probability distribution given the true distribution.
**When to use each?**
In practice, the choice between CrossEntropyLoss and NLLLoss depends on the output of your model. If your model produces class probabilities, it is usually better to use CrossEntropyLoss because it combines both softmax and negative log-likelihood loss. On the other hand, if your model produces log probabilities, then NLLLoss is better suited for this situation.
Another reason to choose NLLLoss is when you have a custom implementation of the output layer and want to avoid applying a softmax function to the outputs.
Overall, when we are optimizing multi-class classification problems with a softmax output layer and our NN is not producing a log probability output, then we should use CrossEntropy Loss. On the other hand, if we make a custom output layer which computes the log probabilities, we should choose the NLL Loss.