LSTM RNN output is just the average of the training data

I have a RNN with a LSTM layer with these options:

layers = [
    sequenceInputLayer(6554)
    lstmLayer(256)
    fullyConnectedLayer(6554)
    regressionLayer];
options = trainingOptions("adam", ...
    MaxEpochs=100, ...
    Shuffle="never", ...
    Plots="training-progress", ...
    Verbose=0);

Each step is a 6554 items-long array, which represents the E-field at a step of the propagation. From this the network should predict the next array, i.e. how the E-field changes from the previous one. I will implement other features but for now it should just be a simple time evolution prediction.

The problem is that I get a constant output from the model, which after checking I realized is just the average of the training data, i.e. each of the 6554 values outputed is the average of all the previous values at the same index in the training data.

Any idea of why this happens and how to fix it?

Thx

Answer 1

[Krishna]

Hello Simone,

The issue you've described, where the model consistently outputs the same value (which happens to be the average of the training data) for all 6554 values, implies that there may be an issue within the training process or data preprocessing pipeline.

Regarding the data, the initial step should involve a thorough examination of its quality.

Following that, it's essential to verify whether the data has been properly normalized and whether the preprocessing steps align with the desired outcome.

For the training, it's advisable to ensure the selection of an appropriate loss function.

Additionally, consider implementing debugging checks within your model to continuously monitor both the model's predictions and gradients throughout the training phase.

Please refer the following documentation to learn more about debugging:

https://in.mathworks.com/help/deeplearning/ug/monitor-deep-learning-training-progress.html