Imbalanced datasets refer to a situation where the number of data points in one class is significantly higher than the other(s). This issue is extremely common in real-world problems, including regression problems. In such a scenario, the machine learning model is likely to be biased towards the majority class and fail to make accurate predictions for the minority class, resulting in poor performance metrics such as low recall/precision. In this context, several techniques have been proposed to handle imbalanced datasets in regression problems, including weighted loss functions and quantile regression.
1. Weighted loss functions- In this technique, we assign higher weights to rare classes and lower weights to common classes. During training, the rare and common classes contribute equally to the overall loss function. Therefore, the model learns to focus on the rare class, which results in better performance for the underrepresented class. The weighted loss function is defined as follows:
$$L(y,\hat{y})=\frac{1}{N}\sum_{i = 1}^{N}w_{i}l(y_{i},\hat{y}_{i})$$
where y is the true label, ŷ is the predicted label, N is the total number of samples, w is the weight assigned to each sample, and L is the loss function. There are different methods of assigning weights, including inverse frequency weighting, where the weight of a class is inversely proportional to its frequency, and synthetic minority oversampling technique (SMOTE), where new samples are generated for the minority class to balance the dataset.
2. Quantile Regression- Another effective technique for handling imbalanced datasets in regression problems is quantile regression (QR). QR estimates the different quantiles of the distribution rather than the mean in the regression problem. It is particularly useful for regression problems that have asymmetric and heteroscedastic distribution. QR models estimate the conditional quantiles of the dependent variable, given the independent variables. Specifically, we fit a separate regression model for different quantile values. For instance, the 50th quantile regression estimates the median or the 2nd quantile, which estimates the point of accumulation of the upper 2
QR is robust in handling outliers and asymmetry, which make it an appropriate tool for imbalanced datasets. Since the model prioritizes underrepresented classes, it results in a better fit for the minority class, reducing the classification bias of the machine learning model.