How to buid a ConvLSTM (not CNN+LSTM) network?

Hello everyone

Is there any way to buid a ConvLSTM network with MATLAB? I have found many examples related to CNN+LSTM, but can't find any example related to ConvLSTM in which the input state and hidden states of LSTM layer are handled with convolutionl filter? The structure can be found in https://proceedings.neurips.cc/paper/2015/file/07563a3fe3bbe7e3ba84431ad9d055af-Paper.pdf.

Answer 1

[Parag]

Hi @Peng Liang,

I understand that you want to build an LSTM network between convolutional operations, rather than placing the LSTM at the end of the convolutional operations.

A ConvLSTM layer is a recurrent layer that utilizes convolution operations internally rather than relying on matrix multiplications. Currently, MATLAB does not support the ConvLSTM layer. However, you can use the following workaround to employ LSTM networks for image classification:

1. Use CNN layers to extract features from the image, ensuring that the final output of the convolution process is a 1-D vector.

2. Feed this 1-D vector into an LSTM, which will produce another 1-D vector as output.

3. Use the output of the LSTM for image classification by passing it through a `classificationLayer`.

MATLAB also provides the capability to define custom layers. You can refer to the MathWorks documentation to learn how to create custom deep learning layers:

https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layers.html

Additionally, to view the list of supported deep learning layers in MATLAB, please visit:

https://www.mathworks.com/help/deeplearning/ug/list-of-deep-learning-layers.html

Hope this helps.