Batch normalization is a popular technique used in deep neural networks (DNNs) to improve training performance. The main idea behind batch normalization is to normalize the input to each layer per batch (i.e., a subset of the training data) during training, so that the distribution of inputs to each layer remains stable.
More specifically, during training, batch normalization computes the mean and variance of each feature in the inputs to a layer over a mini-batch of training examples, and then normalizes the input using these values. The normalized input is then operationally transformed back to the original scale (using scaling and shifting parameters learned during training) before being fed into the next layer.
One of the key advantages of batch normalization is that it can alleviate the "internal covariate shift" problem that arises during training. The internal covariate shift refers to the fact that the distribution of inputs to each layer changes as the weights of the previous layers are updated during training. This makes it harder for the subsequent layers to learn, as they need to adapt to the constantly changing inputs. By normalizing the inputs per batch, however, the distribution of the inputs to each layer remains more stable, which makes it easier for the layers to learn.
Batch normalization also has some additional benefits. It can reduce the dependence of the network on initialization, which can lead to faster training and better accuracy. It can also improve the generalization performance of the network, as it acts as a form of regularization by adding noise to the inputs during training. Finally, it can allow for the use of larger learning rates during training without causing the network to diverge (i.e., allowing for faster convergence to the optimal solution).
To illustrate the benefits of batch normalization consider the following example. Suppose we have a deep neural network with many layers, and during training, the inputs to each layer are always in the same range. This could make training much smoother for each subsequent layer, allowing the network to converge faster to the optimal solution. However, if the input ranges are different per layer, this can cause the gradients going backwards through the network to oscillate or vanish due to the vanishing gradient problem, making training more difficult.
In summary, batch normalization is an effective technique for improving training performance in DNNs by stabilizing the distributions of inputs to each layer, reducing internal covariate shift, and providing additional regularization.