Random forests and gradient boosting machines (GBM) are both ensemble learning methods used for supervised learning tasks such as classification and regression. Ensemble learning uses multiple models to improve the performance of a single model.
Random Forest:
Random forests work by combining multiple decision trees. Each decision tree in the random forest is trained on a different subset of the input data, chosen randomly with replacement, and uses a random subset of the features. When making a prediction, the output of each individual tree is combined to form the final prediction. The goal of this approach is to create a set of overfitting trees with low correlation among themselves, so the combined output has lower variance than a single tree.
The random forest algorithm can be described mathematically as follows,
Let xn be the nth observation and yn be the corresponding target variable, where nβ=β1,β2,β...,βN and N is the sample size. The algorithm first grows B independent trees on separate, random subsets of the data. The predicted value for an observation xn is computed as the average of the predictions from all the individual trees, i.e.
$$f(x_n) = \frac{1}{B}\sum_{b=1}^B f_b(x_n)$$
Gradient Boosting Machine:
Gradient Boosting Machine (GBM) is another ensemble method composed of a sequence of decision trees. In contrast to random forests, however, GBM uses a series of decision trees. In each iteration, a weak tree is added to the model to reduce the errors from the previous iteration. Unlike random forests, each decision tree is trained on the residuals of the previous tree.
The GBM algorithm can also be described mathematically as follows,
Let xn be the nth observation and yn be the corresponding target variable, where nβ=β1,β2,β...,βN and N is the sample size. The algorithm first fits an initial model, f0(x), and creates a series of subsequent models by sequentially applying a weak learner to the residuals of the previous model, creating an additive ensemble of base learners h(x). That is,
$$f_0(x)=\underset{\gamma}{\operatorname{argmin}}\sum_{i=1}^{N}L(y_i, \gamma)$$
fm(x)β=βfmβ
ββ
1(x)β
+β
Ξ³mhm(x)
where Ξ³m is the learning rate for the mth tree, hm(x) is the mth decision tree, and L(y,βf(x)) is the loss function for a given response variable y and a model f(x).
The main difference between random forests and GBM is how the individual trees are trained and combined. Random forests use independent trees grown on separate, random subsets of the data with different feature subsets. GBM, on the other hand, uses a sequential approach where each tree is trained on the residuals of the previous tree. In this way, GBM can achieve higher accuracy than random forests, but it is also more complex and prone to overfitting.