Keras offers a range of activation functions that can be used in neural networks, but sometimes you may need to design and implement a custom activation function that is specific to your problem domain. Here’s how to do it:
To define a custom activation function in Keras, you should define a new class that inherits from Keras’ Activation class. The class must implement two methods: __init__ and __call__. The __init__ method is used to initialize the activation function, while the __call__ method is the one that applies the activation function to the input tensor.
Here’s an example of a custom activation function that applies the exponential function to the input:
from tensorflow.keras.layers import Activation
from tensorflow.keras import backend as K
class ExpActivation(Activation):
def __init__(self, activation, **kwargs):
super(ExpActivation, self).__init__(activation, **kwargs)
self.__name__ = 'exp_activation'
def call(self, inputs):
return K.exp(inputs)Once you’ve defined your custom activation function, you can use it in your Keras model just like any other activation function. For example:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
model = Sequential()
model.add(Dense(64, input_shape=input_shape))
model.add(ExpActivation('exp'))However, before creating a custom activation function, some considerations should be made:
1. **Differentiability**: Most modern deep learning frameworks like Keras make use of automatic differentiation to compute gradients, which is an important aspect of backpropagation. Being able to compute gradients for activation functions is important for optimizers like stochastic gradient descent (SGD) to be able to update network weights effectively. So, it’s crucial that your custom activation function is differentiable.
2. **Monotonicity**: Monotonicity is a mathematical property that says if the input to the function increases, then the output also increases. For example, the ReLU activation function is monotonic since it returns 0 for all negative inputs and linearly increases for all non-negative inputs. Having a monotonic activation function can help speed up convergence during training.
3. **Range of output**: Activation functions output values should be within a certain range. Commonly used ranges are [-1, 1] and [0, 1]. If an activation function’s output values are not within the expected range, the training process may break down.
4. **Computational complexity**: Custom activation functions should be computationally efficient. Highly complex activation functions may increase the processing time per sample, slowing down the training process and making it difficult to scale up to larger datasets.
These considerations are important to keep in mind when designing and implementing custom activation functions to ensure that the resulting network trains efficiently, generalizes well to new data and can be optimized effectively.