Documentation

# batchNormalizationLayer

Batch normalization layer

## Description

A batch normalization layer normalizes each input channel across a mini-batch. To speed up training of convolutional neural networks and reduce the sensitivity to network initialization, use batch normalization layers between convolutional layers and nonlinearities, such as ReLU layers.

The layer first normalizes the activations of each channel by subtracting the mini-batch mean and dividing by the mini-batch standard deviation. Then, the layer shifts the input by a learnable offset β and scales it by a learnable scale factor γ.

## Creation

### Syntax

``layer = batchNormalizationLayer``
``layer = batchNormalizationLayer('Name',Value)``

### Description

````layer = batchNormalizationLayer` creates a batch normalization layer.```

example

````layer = batchNormalizationLayer('Name',Value)` creates a batch normalization layer and sets the optional Batch Normalization, Parameters and Initialization, Learn Rate and Regularization, and `Name` properties using name-value pairs. For example, `batchNormalizationLayer('Name','batchnorm')` creates a batch normalization layer with the name `'batchnorm'`. You can specify multiple name-value pairs. Enclose each property name in single quotes.```

## Properties

expand all

### Batch Normalization

Input mean of each channel, specified as a numeric array of size 1-by-1-by-`NumChannels` for 2-D input or 1-by-1-by-1-by-`NumChannels` for 3-D input.

After network training finishes, the software calculates the input mean over the entire training data set. The layer uses `TrainedMean` (in place of the mini-batch mean) to normalize the input during prediction.

Input variance of each channel, specified as a numeric array of size 1-by-1-by-`NumChannels` for 2-D input or 1-by-1-by-1-by-`NumChannels` for 3-D input.

After network training finishes, the software calculates the input variance over the entire training data set. The layer uses `TrainedVariance` (in place of the mini-batch variance) to normalize the input during prediction.

Constant to add to the mini-batch variances, specified as a numeric scalar equal to or larger than `1e-5`.

The batch normalization layer adds this constant to the mini-batch variances before normalization to ensure numerical stability and avoid division by zero.

Number of input channels, specified as `'auto'` or a positive integer.

This property is always equal to the number of channels of the input to the layer. If `NumChannels` equals `'auto'`, then the software infers the correct value for the number of channels at training time.

### Parameters and Initialization

Function to initialize the channel scale factors, specified as one of the following:

• `'ones'` – Initialize the channel scale factors with ones.

• `'zeros'` – Initialize the channel scale factors with zeros.

• `'narrow-normal'` – Initialize the channel scale factors by independently sampling from a normal distribution with zero mean and standard deviation 0.01.

• Function handle – Initialize the channel scale factors with a custom function. If you specify a function handle, then the function must be of the form ```scale = func(sz)```, where `sz` is the size of the scale. For an example, see Specify Custom Weight Initialization Function.

The layer only initializes the channel scale factors when the `Scale` property is empty.

Data Types: `char` | `string` | `function_handle`

Function to initialize the channel offsets, specified as one of the following:

• `'zeros'` – Initialize the channel offsets with zeros.

• `'ones'` – Initialize the channel offsets with ones.

• `'narrow-normal'` – Initialize the channel offsets by independently sampling from a normal distribution with zero mean and standard deviation 0.01.

• Function handle – Initialize the channel offsets with a custom function. If you specify a function handle, then the function must be of the form ```offset = func(sz)```, where `sz` is the size of the scale. For an example, see Specify Custom Weight Initialization Function.

The layer only initializes the channel offsets when the `Offset` property is empty.

Data Types: `char` | `string` | `function_handle`

Channel scale factors γ, specified as a numeric array.

The channel scale factors are learnable parameters. When training a network, if `Scale` is nonempty, then `trainNetwork` uses the `Scale` property as the initial value. If `Scale` is empty, then `trainNetwork` uses the initializer specified by `ScaleInitializer`.

At training time:

• For 2-D input, `Scale` is a 1-by-1-by-`NumChannels` array of positive numbers.

• For 3-D input, `Scale` is a 1-by-1-by-1-by-`NumChannels` array of positive numbers.

Channel offsets β, specified as a numeric array.

The channel offsets are learnable parameters. When training a network, if `Offset` is nonempty, then `trainNetwork` uses the `Offset` property as the initial value. If `Offset` is empty, then `trainNetwork` uses the initializer specified by `OffsetInitializer`.

At training time:

• For 2-D input, `Offset` is a 1-by-1-by-`NumChannels` array.

• For 3-D input, `Offset` is a 1-by-1-by-1-by-`NumChannels` array.

### Learn Rate and Regularization

Learning rate factor for the scale factors, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the scale factors in a layer. For example, if `ScaleLearnRateFactor` is 2, then the learning rate for the scale factors in the layer is twice the current global learning rate. The software determines the global learning rate based on the settings specified with the `trainingOptions` function.

Learning rate factor for the offsets, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the offsets in a layer. For example, if `OffsetLearnRateFactor` equals 2, then the learning rate for the offsets in the layer is twice the current global learning rate. The software determines the global learning rate based on the settings specified with the `trainingOptions` function.

L2 regularization factor for the scale factors, specified as a nonnegative scalar.

The software multiplies this factor by the global L2 regularization factor to determine the learning rate for the scale factors in a layer. For example, if `ScaleL2Factor` is 2, then the L2 regularization for the offsets in the layer is twice the global L2 regularization factor. You can specify the global L2 regularization factor using the `trainingOptions` function.

L2 regularization factor for the offsets, specified as a nonnegative scalar.

The software multiplies this factor by the global L2 regularization factor to determine the learning rate for the offsets in a layer. For example, if `OffsetL2Factor` is 2, then the L2 regularization for the offsets in the layer is twice the global L2 regularization factor. You can specify the global L2 regularization factor using the `trainingOptions` function.

### Layer

Layer name, specified as a character vector or a string scalar. To include a layer in a layer graph, you must specify a nonempty unique layer name. If you train a series network with the layer and `Name` is set to `''`, then the software automatically assigns a name to the layer at training time.

Data Types: `char` | `string`

Number of inputs of the layer. This layer accepts a single input only.

Data Types: `double`

Input names of the layer. This layer accepts a single input only.

Data Types: `cell`

Number of outputs of the layer. This layer has a single output only.

Data Types: `double`

Output names of the layer. This layer has a single output only.

Data Types: `cell`

## Examples

collapse all

Create a batch normalization layer with the name `'BN1'`.

`layer = batchNormalizationLayer('Name','BN1')`
```layer = BatchNormalizationLayer with properties: Name: 'BN1' NumChannels: 'auto' TrainedMean: [] TrainedVariance: [] Hyperparameters Epsilon: 1.0000e-05 Learnable Parameters Offset: [] Scale: [] Show all properties ```

Include batch normalization layers in a `Layer` array.

```layers = [ imageInputLayer([32 32 3]) convolution2dLayer(3,16,'Padding',1) batchNormalizationLayer reluLayer maxPooling2dLayer(2,'Stride',2) convolution2dLayer(3,32,'Padding',1) batchNormalizationLayer reluLayer fullyConnectedLayer(10) softmaxLayer classificationLayer ]```
```layers = 11x1 Layer array with layers: 1 '' Image Input 32x32x3 images with 'zerocenter' normalization 2 '' Convolution 16 3x3 convolutions with stride [1 1] and padding [1 1 1 1] 3 '' Batch Normalization Batch normalization 4 '' ReLU ReLU 5 '' Max Pooling 2x2 max pooling with stride [2 2] and padding [0 0 0 0] 6 '' Convolution 32 3x3 convolutions with stride [1 1] and padding [1 1 1 1] 7 '' Batch Normalization Batch normalization 8 '' ReLU ReLU 9 '' Fully Connected 10 fully connected layer 10 '' Softmax softmax 11 '' Classification Output crossentropyex ```

expand all

## Algorithms

A batch normalization normalizes its inputs xi by first calculating the mean μB and variance σB2 over a mini-batch and over each input channel. Then, it calculates the normalized activations as

`$\stackrel{^}{{x}_{i}}=\frac{{x}_{i}-{\mu }_{B}}{\sqrt{{\sigma }_{B}^{2}+ϵ}}.$`

Here, ϵ (the property `Epsilon`) improves numerical stability when the mini-batch variance is very small. To allow for the possibility that inputs with zero mean and unit variance are not optimal for the layer that follows the batch normalization layer, the batch normalization layer further shifts and scales the activations as

`${y}_{i}=\gamma {\stackrel{^}{x}}_{i}+\beta .$`

Here, the offset β and scale factor γ (`Offset` and `Scale` properties) are learnable parameters that are updated during network training.

When network training finishes, the batch normalization layer calculates the mean and variance over the full training set and stores them in the `TrainedMean` and `TrainedVariance` properties. When you use a trained network to make predictions on new images, the layer uses the trained mean and variance instead of the mini-batch mean and variance to normalize the activations.

## References

[1] Ioffe, Sergey, and Christian Szegedy. "Batch normalization: Accelerating deep network training by reducing internal covariate shift." preprint, arXiv:1502.03167 (2015).