Hi Ioana,
There are a few modifications to your data and network that are required to make your workflow possible.
The difficulty is associated with the fact that the convolutional part of your network needs to operate on data of the form S x S x C x B (in your case, this is equal to 1 x L x 1 x 91147, where L = 300 to start with, as you rightly pointed out), but the recurrent part of your network (that is, the bilstmLayer) needs data in the format C x B x T (in your case, 1 x 91147 x L, where L is the length of the segment once it's gone through all the convolution layers).
The issue here is that your dimension which I've marked as L is at first a spatial dimension, and then a time dimension. To make this switch possible, we are going to need a custom layer. More on that later - for now, let's start by thinking about how you want to specify your training data.
The sequenceInputLayer requires data in cell array format, as Srivardhan pointed out.
For your training data, you're going to need one cell array for each observation (so 91147 cell arrays), and in each of these cell arrays, you'll need a S x S x C x T array (see this part of the trainNetwork documentation for more details). In your case, these arrays will be of size 1 x 300 x 1 x 1 - the singleton T dimension is irrelevant here, because we've marked the segment length as a spatial dimension, but it'll become relevant later, when we use it in our custom layer. The following line of code shows how I've created random data in this format - you're going to want to get your 1-by-300-by-1-by-91147 data array in this format too:
inputSize = [1 300 1 1];
numTrainSamples = 91147;
xtrain = arrayfun(@(x)rand(inputSize),1:numTrainSamples,'UniformOutput',false)';
For your responses, because each of the 91147 sequences has a single label, you can keep the 91147 x 1 vector of categorical responses.
Now, on to the network. The main changes you need to make are the following:
1) Instead of the flatten layer, you will need to create a custom layer which transforms your S x S x C x B data to C x B x T data. If you are unfamiliar with custom layers, all you need to do is save the code below into its own file called 'imageToSequenceCustomLayer.m' and include this file in the directory with your main network training file:
classdef imageToSequenceCustomLayer < nnet.layer.Layer & nnet.layer.Formattable
methods
function this = imageToSequenceCustomLayer(nanvargsme)
arguments
nvargs.Name (1,:) {mustBeText} = "imageToSequence"
end
this.Name = nvargs.Name;
end
function Y = predict(~, X)
% Permute from S x S x C x B to C x B x T x S
Y = permute(stripdims(X), [3 4 2 1]);
% Re-label the array with the correct dimensions
Y = dlarray(Y, 'CBT');
end
end
end
You can then just replace your flattenLayer with
imageToSequenceCustomLayer()
The flattening is somewhat implicit in the permutation of the data that happens in the customLayer, so we don't need it anymore!
The final thing you'll need to do is to specify the 'OutputMode' NVP of your biltsmLayer as 'last', so that you don't classify every single time step of the sequence. Instead, you output the last time step and use that to classify the entire sequence.
Your network should look as shown below:
layers = [
sequenceInputLayer([1 300 1],'Normalization', 'zscore', 'Name','input');
sequenceFoldingLayer('Name','fold')
convolution2dLayer([1 7], 16,'stride',[1 1], 'padding','same','Name','conv1')
batchNormalizationLayer('Name','bn1')
maxPooling2dLayer([1 2],'stride',[1 2],'Name','mpool1')
convolution2dLayer([1 7], 32,'stride',[1 1], 'padding','same','Name','conv2')
batchNormalizationLayer('Name','bn2')
reluLayer('Name','relu1')
maxPooling2dLayer([1 2],'stride',[1 2],'Name','mpool2')
convolution2dLayer([1 5], 64,'stride',[1 1], 'padding','same','Name','conv3')
batchNormalizationLayer('Name','bn3')
reluLayer('Name','relu2')
convolution2dLayer([1 5], 128,'stride',[1 1], 'padding','same','Name','conv4')
batchNormalizationLayer('Name','bn4')
reluLayer('Name','relu3')
convolution2dLayer([1 3], 256,'stride',[1 1], 'padding','same','Name','conv5')
batchNormalizationLayer('Name','bn5')
reluLayer('Name','relu4')
maxPooling2dLayer([1 2],'stride',[1 2],'Name','mpool3')
convolution2dLayer([1 3], 512,'stride',[1 1], 'padding','same','Name','conv6')
batchNormalizationLayer('Name','bn6')
reluLayer('Name','relu5')
maxPooling2dLayer([1 2],'stride',[1 2],'Name','mpool4')
sequenceUnfoldingLayer('Name','unfold')
imageToSequenceCustomLayer()
bilstmLayer(200,'Name','lstm', 'OutputMode', 'last')
reluLayer('Name','relu6')
fullyConnectedLayer(256,'Name','fc1')
reluLayer('Name','relu7')
fullyConnectedLayer(128,'Name','fc2')
reluLayer('Name','relu8')
fullyConnectedLayer(5,'Name','fc3')
softmaxLayer('Name','softmax')
classificationLayer('Name','classification')
];
This should not throw any errors!
Hope this helps!
Tomaso
