Class imbalance is a common issue in machine learning where the number of instances for one class greatly exceeds the number of instances for another class. This can lead to a biased model that performs poorly on the minority class. There are several techniques for handling class imbalance in machine learning, including oversampling, undersampling, and cost-sensitive learning.
1. Oversampling: Oversampling involves increasing the number of instances in the minority class to balance the classes. This is done by either duplicating existing instances or generating new synthetic instances. There are several algorithms for generating synthetic instances, such as Synthetic Minority Over-sampling Technique (SMOTE), Adaptive Synthetic Sampling (ADASYN), and Random Minority Over-sampling (RAMO). SMOTE, for example, creates synthetic instances by interpolating between existing instances in the minority class.
2. Undersampling: Undersampling involves reducing the number of instances in the majority class to balance the classes. This is done by either randomly removing instances or selecting instances based on some criterion. Random undersampling removes instances randomly from the majority class, whereas informed undersampling selects instances based on some criterion such as the distance from the decision boundary.
3. Cost-sensitive learning: Cost-sensitive learning involves assigning different costs to misclassifications for each class. This is done by adjusting the class weights or by using alternative cost functions. Class weights can be adjusted to increase the penalty for misclassifying instances in the minority class. Alternative cost functions such as the F-measure can also be used to account for the imbalance in the data.
Let’s take an example where we’re trying to predict whether a bank loan would be approved or not based on the applicant’s characteristics. Suppose out of 1000 loan applications, only 100 were rejected. In this case, we have an imbalanced dataset where the minority class is the "rejected" class. To handle this imbalanced dataset, we can use any of the above-mentioned techniques.
For example, we can use SMOTE to generate synthetic instances of the rejected class to balance the dataset. This will increase the number of instances of the rejected class, which will prevent the model from being biased towards the "approved" class. Alternatively, we can use cost-sensitive learning to increase the penalty for misclassifying instances of the rejected class. This will also prevent the model from being biased towards the "approved" class.
It’s important to note that each technique has its advantages and disadvantages, and the choice of technique depends on the specific dataset and problem at hand.