Implementing CNN and LSTM in parallel

How to implment a CNN to work for image classifaction and in parallel with LSTM to classify the signals?

Answer 1

[Krishna]

Hello Mahmoud,

To achieve image classification with a CNN and concurrently classify signals with an LSTM, you can employ deep learning networks. Here's a structured approach:

Construct the CNN for Image Classification, build a sequence of convolutional layers with activation functions like ReLU to process images and extract their features.

Develop the LSTM Network for Signal Classification, initiate with an input layer tailored for sequential data such as signals. Insert LSTM layers to handle the time-series data and identify time-related patterns. Post-LSTM, add dense layers to make sense of the extracted sequence features.

Merge the CNN and LSTM for Concurrent Processing, Forge two distinct input pathways, one for the CNN (handling images) and another for the LSTM (processing signals). Each pathway should analyse its input and extract relevant features. Fuse the outputs from both pathways to add the feature sets. If necessary, introduce more dense layers to effectively blend the features for the classification task. To add 2 layers together you’ll need to use ‘additionLayer’. Please go through this example to know how to do this,

https://in.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.additionlayer.html

After you can follow the normal procedure of Neural Networks training like defining optimizers and all. For a practical example of integrating CNN with LSTM, you can refer to this resource:

https://www.mathworks.com/help/deeplearning/ug/sequence-classification-using-cnn-lstm-network.html

Additionally, to deepen your understanding of the layers and their functions,

https://in.mathworks.com/help/deeplearning/ug/list-of-deep-learning-layers.html

Hope this helps.

[MAHMOUD EID]

Thank you for reply. The problem not in how to create the layers that represent the network. The challenge is how to feed the images and signals for the classifcation to this network architecture

[Krishna]

Can you explain me in more detail the problem statement what you are trying to achieve. Then maybe I can help you.

[MAHMOUD EID]

I have an images with 12 classes and the same signals of these images also has the same number of classes. I need to classify the images using 2D CNN and the signals using 1-D CNN and then concentrate the decisions of the two networks together to calssify the signals.

[Venu]

You can use imageDatastore for handling images, signalDatastore or arrayDatastore for signals and you can combine those data stores to pass through trainNetwork