Gradient descent is an iterative optimization algorithm used to find the minimum of a cost function. The difference between online and batch gradient descent is the amount of data used to update the parameters in each iteration.
**Batch Gradient Descent:** In batch gradient descent, we use the entire dataset to calculate the gradient of the cost function with respect to the model parameters. Then, we update the parameters based on the average of the gradients over all the examples in the dataset. The formula for updating the parameters using batch gradient descent is:
$$\theta_{j} = \theta_{j} - \alpha \frac{1}{m} \sum_{i=1}^{m}(h_{\theta}(x^{(i)}) - y^{(i)})x_j^{(i)}$$
where θj is the jth model parameter, α is the learning rate, m is the number of examples in the dataset, hθ(x(i)) is the predicted output for the ith example, y(i) is the actual output for the ith example, and xj(i) is the value of the jth feature for the ith example.
Batch gradient descent can be slow and computationally expensive on large datasets since we need to calculate the gradients for all the examples in the dataset before updating the parameters. However, batch gradient descent can converge to the global minimum of the cost function, unlike stochastic gradient descent.
**Stochastic Gradient Descent:** In stochastic gradient descent, we randomly select one example from the dataset and use it to update the model parameters. The formula for updating the parameters using stochastic gradient descent is:
θj = θj − α(hθ(x(i)) − y(i))xj(i)
where θj, α, hθ(x(i)), y(i), and xj(i) are the same as in batch gradient descent.
Stochastic gradient descent updates the parameters more frequently than batch gradient descent, and each update is based on a single example. This can make stochastic gradient descent converge faster than batch gradient descent, especially on large datasets.
**Mini-Batch Gradient Descent:** Mini-batch gradient descent is a compromise between batch and stochastic gradient descent, where we randomly select a small subset (or minibatch) of the data to calculate the gradients, and then update the parameters based on the average of the gradients over the minibatch. This combines the benefits of both batch and stochastic gradient descent and can converge faster than batch gradient descent while being less noisy than stochastic gradient descent.
In general, batch gradient descent is used when the dataset can fit into memory, and when we have a high tolerance for computation time. Stochastic gradient descent is used when we have a large dataset, and when we want to update the parameters frequently. Mini-batch gradient descent is often used as a compromise between the two when the dataset is too large for batch gradient descent but small enough to fit into memory in batches.