Training Network with Sequence input of different Nature

Hello,

I am curently working on an IA project in which I have inputs of different kind in input of my network. For exemple with 2 inputs, one is a sensor value and the other is an int value with another information.

Until now I was working with features inputs and I had one feature input for each inputs of my network (in the exemple my network had 2 feature inputs).

Now, I want to use sequences in input (one sequence of sensor values and one sequence of int values). However when I declare 2 sequence inputs with the network designer, the verification process tells me that I can't use multiple sequence input.

I looked the following topics but I did not find something that seem to match with my case.

• https://fr.mathworks.com/help/deeplearning/ug/classify-sequence-data-using-lstm-networks.html
• https://fr.mathworks.com/help/deeplearning/ug/multiple-sequences-with-dynamic-neural-networks.html
• https://fr.mathworks.com/help/deeplearning/ug/time-series-forecasting-using-deep-learning.html

Do you have any idea to help me on thios or should I create a custom sequence input layer ?

Thanks in advance

Mathieu

Answer 1

[Milan Bansal]

Hi Mathieu, 

As per my understanding you are trying to build a neural network with multiple inputs (two different sequences) and facing an error while declaring two sequence input layers in the network.

A neural network in MATLAB cannot have two input layers in it. To build a neural network with multiple inputs, it is required to create the network architecture using "layerGraph" object. 

Please refer to the documentation below to learn more about "layerGraph" object. 

https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layergraph.html 

Please refer to the steps below for building a neural network with multiple sequence inputs: -

1. Initially, define the main architecture of the neural network with "sequenceInputLayer" as input layer and other desired layers.
2. Add a "concatenationLayer" in the network with the dimensions such that the second "sequenceInputLayer" can be connected to the main architecture later for another sequence input.
3. Convert the network into Layer Graph. 
4. Create a "sequenceInputLayer" for sequence input and connect it to the second input of the "concatenationLayer". 

Please refer to the pseudo code in the code snippet below: -

layers = [ ... 
    sequenceInputLayer(inputSize) 
    lstmLayer(numHiddenUnits,'OutputMode','last') 
    fullyConnectedLayer(numClasses) 
    softmaxLayer 
    classificationLayer] 
 
lgraph = layerGraph(layers);  
seqInput = sequenceInputLayer(inputSize);  
lgraph = addLayers(lgraph,,'Name','seq');  
lgraph = connectLayers(lgraph,"seq","cat/in2");

Please refer to the example in the documentation below to learn more about Training Neural Network with Multiple Inputs. 

https://www.mathworks.com/help/deeplearning/ug/train-network-on-image-and-feature-data.html  

Hope it helps!

[Pascal]

Hi Milan,

while using multiple Inputs and combinations thereof, such as Images and Features (and Sequences with Features) works fine, I get the following errors when trying to use multiple Sequences Inputs:

Error received when using multiple Sequence Input Layers in Deep NEtwork Designer:

Error received when following your example and trying to manually add the second sequence input to the layer graph:

Is there any workaround for this problem, so that I can use multiple Sequence Inputs in the Network in Matlab?