Batch gradient descent, mini-batch gradient descent, and stochastic gradient descent are optimization methods used for training machine learning algorithms. They differ in the number of samples used to calculate the gradient of the objective function and update the model’s parameters at each iteration. In all three methods, the goal is to minimize the cost function or error function, which measures the distance between the predicted value and the actual value.
Batch gradient descent computes the gradient of the cost function with respect to the parameters of the model for the entire training set. It means we update the model parameters after processing the entire dataset. The formula for batch gradient descent is:
$\theta = \theta - \alpha \frac{1}{m}\sum_{i=1}^{m} \nabla J(\theta;x^{(i)},y^{(i)})$
Where, θ is the parameter vector, m is the number of instances in the training set, α is the learning rate, and J(θ; x(i), y(i)) is the cost function for an individual training example.
Batch gradient descent has the advantage of being guaranteed to converge to the global minima for a convex cost function while being slow to compute the gradient and update the model parameters due to the large amount of data involved.
Stochastic gradient descent, on the contrary, processes each training example one by one and updates the model’s parameters with each example. The formula for stochastic gradient descent is:
θ = θ − α∇J(θ; x(i), y(i))
Where, i is the current example being processed.
Stochastic gradient descent has the advantage of being computationally cheap and can jump out of local minima. It, however, has a high variance and requires a small learning rate to prevent the parameter updates from oscillating wildly.
Mini-batch gradient descent is a variation of stochastic gradient descent that processes a small batch of samples at each iteration instead of one single instance. The batch size is usually between tens to hundreds of examples. The formula for mini-batch gradient descent is:
$\theta = \theta - \alpha \frac{1}{b}\sum_{i=1}^{b} \nabla J(\theta;x^{(i)},y^{(i)})$
Where, b is the batch size, and i refers to the current example in the batch.
Mini-batch gradient descent combines the advantages of both batch and stochastic gradient descent. It has lower variance than stochastic gradient descent, more stable convergence than batch gradient descent since convergence is reached faster, and is more efficient than batch gradient descent because of the decrease in the number of iterations required to reach convergence.
In summary, the main differences between stochastic gradient descent, mini-batch gradient descent, and batch gradient descent are as follows:
- Stochastic gradient descent updates the model parameters after processing each training example, while batch gradient descent updates the parameters once after processing the entire dataset.
- Mini-batch gradient descent processes a small subset of data at each iteration, usually containing several tens to hundreds of samples.
- Batch gradient descent has a lower variance but is slower to compute the gradient and update the model parameters. Stochastic gradient descent and mini-batch gradient descent have higher variance but are more computationally efficient.