Gradient descent is a popular optimization algorithm used in machine learning to minimize the loss function of a model. There are three main variants of gradient descent: batch gradient descent, stochastic gradient descent, and mini-batch gradient descent.
Batch Gradient Descent: Batch gradient descent computes the gradients of the entire training dataset with respect to the model parameters and updates the parameters based on the average gradient. This approach guarantees convergence to the global minimum of the loss function, but can be computationally expensive, especially for large datasets.
Stochastic Gradient Descent: Stochastic gradient descent (SGD) updates the model parameters based on the gradient of a single training example at a time. This approach is much faster than batch gradient descent, but can lead to noisy updates and slow convergence. To mitigate this issue, the learning rate is typically reduced over time.
import tensorflow as tf
# create a stochastic gradient descent optimizer
optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)
# compile the model with the optimizer
model.compile(optimizer=optimizer, loss='mse')
# train the model with stochastic gradient descent
model.fit(x_train, y_train, epochs=100, batch_size=1)Mini-Batch Gradient Descent: Mini-batch gradient descent is a compromise between batch gradient descent and stochastic gradient descent. It computes the gradients of a small batch of training examples at a time and updates the parameters based on the average gradient. This approach is faster than batch gradient descent and less noisy than stochastic gradient descent. The batch size is typically chosen to be a power of 2.
import tensorflow as tf
# create a mini-batch gradient descent optimizer
optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)
# compile the model with the optimizer
model.compile(optimizer=optimizer, loss='mse')
# train the model with mini-batch gradient descent
model.fit(x_train, y_train, epochs=100, batch_size=32)In summary, batch gradient descent computes the gradients of the entire training dataset, while stochastic gradient descent and mini-batch gradient descent compute the gradients of a single training example and a small batch of training examples, respectively. Stochastic gradient descent and mini-batch gradient descent are faster than batch gradient descent, but can be more noisy and require more hyperparameter tuning.