Normalization Layer

Note

We will refer to normalization layers as Norm layers.

What do Norm layers do?

Norm layers normalize the input. Normalization really helps stabilize training and improve training speed. Before norm layers are introduced, people had a very difficult time training huge models (big and deep) because of exploding/vanishing gradients, and normalization simply removes that issue mostly.

When to use Norm layers?

Almost always. If you want to train a big and deep neural network, remember to normalize your input.