In Machine Learning, we develop models to make predictions or classifications on new data. The goal of a loss function is to quantify how well our model is performing these tasks. It measures the discrepancy between predicted outputs and their respective true outputs. The objective is to minimize this discrepancy, which is the reason why loss functions are also referred to as cost functions or error functions.
There is no one-size-fits-all loss function applicable to every problem. The appropriate selection of a loss function depends on the problem at hand, type of data, and the approach used to tackle the problem. Here are some common examples of loss functions widely used in Machine Learning:
**1. Mean Squared Error (MSE) loss function:**
$$MSE = \frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2$$
This loss function is utilized when dealing with regression problems. It computes the average squared distance between the predicted and true values. Squaring the difference has the effect of making large deviations more impactful than smaller ones. As a result, MSE helps the optimization process focus on effective predictions rather than just directional accuracy. One of the limitations of MSE is that it is susceptible to small outliers in the data.
**2. Binary Cross Entropy loss function:**
$$BCE = -\frac{1}{n} \sum_{i=1}^{n} y_i log(p) + (1-y_i) log(1-p)$$
The binary cross-entropy (BCE) loss function is commonly deployed in binary classification problems. It calculates the average difference between the predicted and actual binary classification labels. BCE measures the probability distribution between two classes by computing the cross-entropy over each probability. This loss function is well-suited for problems with highly imbalanced data.
**3. Categorical Cross-Entropy loss function:**
$$CCE = -\frac{1}{n}\sum_{i=1}^{n} \sum_{j=1}^{m} y_{ij} log(p_{ij})$$
Categorical Cross-Entropy (CCE) loss function is a prominent choice when working with multiclass classification problems. It determines the difference between the predicted class probabilities and the true class probabilities. When dealing with CCE, the label must take a one-hot encoding approach. The loss function is the sum of log probability of each category weighted by the actual probability distribution.
**4. Hinge Loss function:**
HLβ=βmax(0,β1β
ββ
yiβ
Γβ
yΜi)
Hinge Loss (HL) function is the prevalent choice in Support Vector Machines (SVM) systems. This loss function is used to optimize classification problems. HL subtracts the product of actual and predicted values from 1 and then takes the maximum between 0 and this value. It aims to maximize the margin between positive and negative samples.
In conclusion, loss functions play a vital role in evaluating the performance of the Machine Learning models. Selecting an appropriate loss function for the problem in hand can significantly impact the modelβs accuracy, and the speed at which it trains.