Linear regression is a widely used technique for modelling the relationship between a dependent variable and one or more independent variables. In linear regression, we try to find a line (or a hyperplane in higher dimensions) that best fits the data points. However, sometimes the linear model may lead to overfitting or underfitting, which can result in poor prediction accuracy. Regularization is a technique used to prevent overfitting by adding a penalty term to the linear regression objective function, which shrinks the magnitude of the coefficients towards zero.
L1 (Lasso) and L2 (Ridge) regularization are two common techniques used in linear regression to prevent overfitting. In L1 regularization, we add the sum of the absolute values of the coefficients as the penalty term, while in L2 regularization, we add the sum of the squared values of the coefficients as the penalty term.
The L1 regularization (Lasso) can be expressed as:
$$\min_{w}||Xw-y||_{2}^{2}+\alpha\sum_{j=1}^{p}|w_{j}|$$
where w is the coefficient vector, X is the feature matrix, y is the target variable, p is the number of features, and α is the regularization strength hyperparameter.
The L2 regularization (Ridge) can be expressed as:
$$\min_{w}||Xw-y||_{2}^{2}+\alpha\sum_{j=1}^{p}w_{j}^{2}$$
where all terms are the same as in L1 regularization, except the penalty term is the sum of the squared values of the coefficients.
L1 regularization has the property that it forces some of the coefficients to be exactly equal to zero, effectively performing feature selection. This can simplify the model and reduce the risk of overfitting, but also results in a sparser model with fewer coefficients. L2 regularization shrinks the magnitude of all the coefficients towards zero, but none of them become exactly zero.
The choice between L1 and L2 regularization depends on the problem and the nature of the data. If we believe that only a small number of features are relevant for the prediction, then L1 regularization may be better suited. If we believe that all the features may be relevant but are possibly correlated with each other, then L2 regularization may be better suited. There are also other regularizations techniques that can be used, such as Elastic Net, which combines both L1 and L2 regularization.