Hyperparameter tuning is a crucial component of building a deep learning model. Deep learning models contain a large number of hyperparameters that can dramatically impact their performance. There are several ways to perform hyperparameter tuning in Keras to achieve optimal performance, and here are some common ones:
1. Grid Search: One of the most popular methods of hyperparameter tuning is by using a grid search approach. In this method, you set up a grid with different hyperparameter combinations, and then test the performance of the model with each combination. You can use scikit-learn’s ‘GridSearchCV‘ function to do this.
For example, let’s say you’re training a convolution neural network (CNN) for image classification, and you want to find the optimal values for the learning rate, batch size and number of epochs. You can create a grid search object like this:
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import GridSearchCV
from keras.models import Sequential
from keras.layers import Dense, Conv2D, Flatten
def create_model(lr=0.001, batch_size=32):
model = Sequential()
model.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=(28, 28, 1)))
model.add(Flatten())
model.add(Dense(10, activation='softmax'))
model.compile(optimizer=Adam(lr=lr), loss='categorical_crossentropy', metrics=['accuracy'])
return model
# define the grid search parameters
param_grid = {'lr': [0.001, 0.01, 0.1],
'batch_size': [16, 32, 64],
'epochs': [10, 20, 30]}
# create the model
model = KerasClassifier(build_fn=create_model, verbose=0)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3)
grid_result = grid.fit(X_train, y_train)
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))This code creates a CNN model, and a grid search object with different values for learning rate, batch size and number epochs as defined in ‘param_grid‘ dictionary. We use KerasClassifier to wrap the model so we can use the ‘GridSearchCV‘ function from scikit-learn. Finally, the grid search is executed on the dataset with ‘X_train‘ and ‘y_train‘ and returns the best hyperparameters combination and its corresponding performance.
2. Random Search: Another approach is to perform a random search, which searches the hyperparameters space randomly instead of exhaustively. This method can potentially result in better performance than grid search because it allows for a more extensive search of the hyperparameters space and is often more computationally efficient. You can use scikit-learn’s ‘RandomizedSearchCV‘ function to perform a random search.
from sklearn.model_selection import RandomizedSearchCV
# define the grid search parameters
param_grid = {'lr': [0.001, 0.01, 0.1],
'batch_size': [16, 32, 64],
'epochs': [10, 20, 30]}
# create the model
model = KerasClassifier(build_fn=create_model, verbose=0)
random_search = RandomizedSearchCV(estimator=model, param_distributions=param_grid, n_iter=100, cv=3, verbose=2, random_state=42, n_jobs=-1)
random_search.fit(X_train, y_train)
print("Best: %f using %s" % (random_search.best_score_, random_search.best_params_))This code is similar to the previous example, but the ‘GridSearchCV‘ function has been replaced with ‘RandomizedSearchCV‘ function. You can also notice that ‘n_iter‘ is added, which denotes the number of random combinations to try.
3. Bayesian optimization: This approach is more advanced and typically takes longer to compute while yielding better results than Grid or Random search. It involves creating a probability model that can predict the performance of the model with different hyperparameters configurations without actually running it. The probability model is then used to sequentially suggest new combinations of hyperparameters based on the expected improvement, resulting in better results. There are several libraries in Python that can help implement such optimization like hyperopt, Optuna, Keras Tuner, or scikit-optimize, but Keras Tuner provides an easy-to-use interface for tuning Keras models.
Here’s an example of how to use Keras Tuner, which takes care of using different Bayesian optimization methods, to tune a model.
!pip install keras-tuner
import keras_tuner as kt
def model_builder(hp):
model = Sequential()
model.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=(28, 28, 1)))
model.add(Flatten())
model.add(Dense(10, activation='softmax'))
# Tune the number of units of the first Dense layer
# Choose from 32, 64, 128, or 256
hp_units = hp.Choice('units', values=[32, 64, 128, 256])
model.add(Dense(units=hp_units, activation='relu'))
# Tune the learning rate for the optimizer
# Choose from 0.0001, 0.001, 0.01, or 0.1
hp_learning_rate = hp.Choice('learning_rate', values=[1e-4, 1e-3, 1e-2, 1e-1])
model.compile(optimizer=Adam(lr=hp_learning_rate), loss='categorical_crossentropy', metrics=['accuracy'])
return model
tuner = kt.Hyperband(model_builder,
objective='val_accuracy',
max_epochs=10,
factor=3,
directory='my_dir',
project_name='my_project')
tuner.search(X_train, y_train, epochs=10, validation_data=(X_test, y_test))
best_hps = tuner.get_best_hyperparameters(num_trials = 1)[0]
print(f"The optimal number of units is {best_hps.get('units')} and the optimal learning rate for the optimizer is {best_hps.get('learning_rate')}")This code builds a classifier model, but it tunes the number of units of the first dense layer and the learning rate of the optimizer instead of more common hyperparameters. The ‘model_builder‘ function defines a model with tunable hyperparameters. Keras Tuner’s ‘Hyperband‘ class is used to perform hyperparameter search. It uses asymmetric training and early stopping to search asynchronously through the space of models.
In summary, hyperparameter tuning is an iterative process that requires experimentation and patience. By using the mentioned methods, you can find the best possible hyperparameters combination to improve the performance of your model.