The purpose of regularization in machine learning is to prevent overfitting, which occurs when a model learns too much from the training data and cannot generalize well to new, unseen data. This is a common problem in machine learning, where the goal is to create models that can make accurate predictions on new, unseen data, not just the data used for training.
Regularization techniques achieve this goal by adding a penalty term to the objective function being optimized by the model. This penalty term encourages the model to have smaller weights or fewer degrees of freedom, which reduces the model’s tendency to overfit.
Some common regularization techniques include:
1. L1 Regularization (Lasso Regression): Also known as L1 regularization, Lasso regression adds a penalty term to the loss function that is proportional to the absolute value of the weights of the model. This technique encourages the model to have sparse weights or fewer features selected, effectively performing feature selection alongside regularization.
The formula for L1 regularization is:
$\text{Lasso regression loss} = \text{OLS loss} + \lambda\sum_{i=1}^p|\beta_i|$
where OLS loss refers to the mean squared error loss function without regularization, λ is the regularization parameter and β are the weights being estimated.
2. L2 Regularization (Ridge Regression): L2 regularization, also known as ridge regression, adds a penalty term to the loss function that is proportional to the square of the weights of the model. This technique encourages the model to have smaller weights and reduces the impact of outliers in the data.
The formula for L2 regularization is:
$$\text{Ridge regression loss} = \text{OLS loss} + \lambda\sum_{i=1}^p\beta_i^2$$
where OLS loss, λ and β are defined as before.
3. Elastic Net Regularization: Elastic Net regularization is a combination of L1 and L2 regularization. The penalty term in elastic net is a weighted sum of the L1 and L2 norms of the weights, which allows for both sparsity and smoothness in the model.
The formula for elastic net regularization is:
$$\text{Elastic Net loss} = \text{OLS loss} + \lambda_1\sum_{i=1}^p|\beta_i|+\lambda_2\sum_{i=1}^p\beta_i^2$$
where OLS loss, λ1, λ2, and β are defined as before.
By using these regularization techniques, we can prevent overfitting and improve the accuracy and generalizability of our machine learning models.