Controlling Variability in LSTM Training with Dropout in MATLAB

Hi,

I am training an LSTM network in MATLAB that includes both Dropout and BatchNormalization layers. To ensure reproducibility, I have controlled the training-validation split using the rng() function and set a fixed seed just before calling trainNetwork().

However, I still observe variability in results each time I repeat the experiment. To investigate further, I ran the training on CPU and without the Dropout layer, which produced consistent results.

This leads me to suspect that the Dropout layer might be introducing the variability. Is there a way to control which neurons are dropped during training?

Additionally, I would appreciate insights on the expected level of variability in deep learning results and whether my observations align with typical behavior.

Any suggestions or guidance would be greatly appreciated.

Thanks in advance!

[Jayanti]

Could you please provide the code and dataset you used?

Answer 1

[Aravind]

Hi @Vishnu Keyen,

The variability you are seeing in your LSTM network training is indeed likely due to the Dropout layer. Dropout is designed to introduce randomness by randomly setting a fraction of the input units to zero at each update during training, which helps prevent overfitting.

The fixed seed you use with the “rng” function controls overall randomness in your script, but not specifically in the dropout layer. Unfortunately, in MATLAB's Dropout layer, there is no direct way to control which neurons are dropped, as the dropout mask is generated randomly at each iteration. You can learn more about this in the “Algorithms” section of the dropout layer documentation.

Variability in deep learning results is common and can be attributed to factors like random initialization of weights, the stochastic nature of optimization algorithms, and layers like Dropout that introduce randomness. Even with a fixed random seed, some variability can still occur due to these inherent stochastic processes.

Since the purpose of the dropout layer is to introduce randomness into the training process, trying to control which neurons are dropped might not align with its intended use. However, if you wish to control which neurons are dropped during training, you could consider creating a custom dropout layer. This would allow you to define your own logic for dropping neurons. For more information on creating a custom deep learning layer in MATLAB, you can refer to: https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layers.html.

I hope this helps address your questions.