Multi-input imagedatastore

I am trying to train a network with N inputs to perform binary classification. Each image is multispectral (more than 3 channels rgb). I read in a comment from mathworks staff which creates data and stores it in a datastore to train a network with two inputs: image and vector data.

I have also looked at the example Image Classification using CNN with Multi Input, which uses two inuts to classify digits.

Both of these examples do not use imageDatastore. I was wondering if we could use

combine

to combine the imageDatastore for each data sample. I read in Datastores for Deep Learning that the datastore must be a combined or transformed datastore that returns a cell array with (numInputs+1) columns containing the predictors and the responses, where numInputs is the number of network inputs and numResponses is the number of responses.

My question is the following. Can I have multiple imageDatastore objects and combine them? If so, how do I store the label column? I tried the code below and got an error.

imds1 = imageDatastore(...);
imds2 = imageDatastore(...);
labels = ???
datastore = combine(imds1, imds2, labels);

Usually, we can assign the label through imds1.Labels. I also want to know if once combined, data augmentation can be done on the images and random split for training and validation.

Answer 1

[Mohammad Sami]

You can save the labels as a text file and then create a tabularTextDatastore to read them back.

Something like this could work.

read_size = 4; % readsize must be fixed otherwise concatenation will fail

digitDatasetPath = fullfile(matlabroot,'toolbox','nnet', ...
'nndemos','nndatasets','DigitDataset');
imds = imageDatastore(digitDatasetPath, ...
'IncludeSubfolders',true, ...
'LabelSource','foldernames');
imds.ReadSize = read_size;

labels = imds.Labels; % your labels here
% the order of your images and label must be the same
writematrix(labels,'labels.txt');
% %C = categorical
labelStore = tabularTextDatastore('labels.txt','TextscanFormats','%C',"ReadVariableNames",false);
labelStore.ReadSize = read_size;
labelStoreCell = transform(labelStore,@setcat_and_table_to_cell);

finalStore = combine(imds,labelStoreCell);

% test read
finalStore.read

You will also need to correct the categorical array categories during transformation.

function [dataout] = setcat_and_table_to_cell(datain)
validcats = string(0:9); % define valid labels for categorical array
datain.(1) = setcats(datain.(1),validcats);
dataout = table2cell(datain);
end

[OJ27]

thanks, this works. In terms of splitting and data augmentation, how would you go about it? sptitting and augmenting first and then combining?

splitEachLabel;
augmentedImageDatastore;

only accept imageDatastores as input.

[Mohammad Sami]

Yes I think you will have to split before you create your labels data store. And likely you have to augment before you combine.

[OJ27]

I split the imagedatastore before the combine operation, but after all that work I got the message:

Error using trainNetwork (line 170)
Multi-input networks do not support validation.

Caused by:
    Error using trainNetwork>iValidateNoMISOAndValidationData (line 581)
    Multi-input networks do not support validation. 

I guess I will have to code the network with python.

[OJ27]

actually the readsize property is the number of image files to read in a call to the read function. Thus, when read_size =4, 4 images are loaded.

Also, your solution combines one imagedatstore with the labels. I need to combine multiple imagedatastores. An example that comes to mind would be to use tiny imagenet (64 x 64 x 3) and a grayscale version of the same network.

imds1 -> tiny imagenet (64 x 64 x 3)

imds2-> grayscale version (64 x 64 x 1)

label-> 1

{64×64×3 uint8} {64×64 uint8} {[1]}

the error now is

Error using trainNetwork (line 170)
Dimensions of arrays being concatenated are not consistent.

Error in small_multi_input_network (line 93)
[net, info] = trainNetwork(train_multi,lgraph,options);

Caused by:
    Error using cat
    Dimensions of arrays being concatenated are not consistent.

[Mohammad Sami]

That is why i set the readsize to be the same for all underlying image datastore and label datastore.

Otherwise it complains about the arrays being concatenated not being consistent.

You should try setting it to one.

The solution illustrates a way to combine the datastore. you can simply add more image datastores, should still work the same way, as long as the labels are the same.

finalStore = combine(imds,imds2,imdsn,labelStoreCell);

[OJ27]

it doesn't work with read_size = 1. Perhaps it is my cnn? My second imagedatastore is grayscale but adding imdsTrain as the second datastore in the combine just as an example.

% Read tiny images train
imdsTrain = imageDatastore('E:/tiny-imagenet-200/train/', ...
'IncludeSubfolders',true,'FileExtensions','.jpeg');

labels = split(imdsTrain.Files,'E:\tiny-imagenet-200\train\');
labels = split(labels(:,2),'\images');
labels = labels(:,1);
labels = categorical(labels);  
imdsTrain.Labels = labels;

read_size = 1;
imdsTrain.ReadSize = read_size;

% Combine
trainlabels = imdsTrain.Labels; % your labels here
writematrix(trainlabels,'trainlabels.txt');
labelStore = tabularTextDatastore('trainlabels.txt','TextscanFormats','%C',"ReadVariableNames",false);
labelStore.ReadSize = read_size;
labelStoreCell = transform(labelStore,@setcat_and_table_to_cell);
train_multi = combine(imdsTrain,imdsTrain,labelStoreCell);
train_multi.read

% layerGraph
layers1 = createLayer('_1',3)
layers2 = createLayer('_2',3)

concat = depthConcatenationLayer(2,'Name','concat_1');
layers1 = [layers1; concat; convolution2dLayer([3 3],64,"Name","conv_6","Padding","same");...
        fullyConnectedLayer(400,"Name","fc_3");...
        fullyConnectedLayer(200,"Name","fc_4");...
        softmaxLayer('Name','softmax');...
        classificationLayer('Name','classOutput')]
plot(layerGraph(layers1))
lgraph = layerGraph(layers1);
lgraph = addLayers(lgraph,layers2)
lgraph = connectLayers(lgraph,'relu_5_2','concat_1/in2');
plot(lgraph)

%%
options = trainingOptions('adam', ...
    'InitialLearnRate',0.005, ...
    'LearnRateSchedule','piecewise',...
    'MaxEpochs',100, ...
    'MiniBatchSize',1024, ...
    'Verbose',1, ...
    'Plots','training-progress',...
    'Shuffle','never');

[net, info] = trainNetwork(train_multi,lgraph,options);

%%
function layers=createLayer(s, channels)
layers = [
    imageInputLayer([64 64 channels],"Name",strcat("imageinput",s))
    convolution2dLayer([3 3],8,"Name",strcat("conv_1",s),"Padding","same")
    batchNormalizationLayer("Name",strcat("batchnorm_1",s))
    reluLayer("Name",strcat("relu_1",s))
    maxPooling2dLayer([2 2],"Name",strcat("maxpool_1",s),"Stride",[2 2])
    
    convolution2dLayer([3 3],16,"Name",strcat("conv_2",s),"Padding","same")
    batchNormalizationLayer("Name",strcat("batchnorm_2",s))
    reluLayer("Name",strcat("relu_2",s))
    maxPooling2dLayer([2 2],"Name",strcat("maxpool_2",s),"Stride",[2 2])
    
    convolution2dLayer([3 3],32,"Name",strcat("conv_3",s),"Padding","same")
    batchNormalizationLayer("Name",strcat("batchnorm_3",s))
    reluLayer("Name",strcat("relu_3",s))
    maxPooling2dLayer([2 2],"Name",strcat("maxpool_3",s),"Stride",[2 2])
    
    convolution2dLayer([3 3],64,"Name",strcat("conv_4",s),"Padding","same")
    batchNormalizationLayer("Name",strcat("batchnorm_4",s))
    reluLayer("Name",strcat("relu_4",s))
    maxPooling2dLayer([2 2],"Name",strcat("maxpool_4",s),"Stride",[2 2])
    
    convolution2dLayer([3 3],64,"Name",strcat("conv_5",s),"Padding","same")
    batchNormalizationLayer("Name",strcat("batchnorm_5",s))
    reluLayer("Name",strcat("relu_5",s))]
end

[Mohammad Sami]

I see you are using the same data store twice in the combine. This may be causing the issue. If you want to use the same images, perhaps make a second instance of the data store.

[Girish Tiwari]

Thanks M. Sami. Works like a charm.

However, I am getting an error while training the network with the datastore as "The output size (7) of the last layer does not match the number of classes (10)".

I have verified that there are only 7 labels and fullyconnected layer also has 7 outputs.

(See my code at: https://www.mathworks.com/matlabcentral/answers/619943-invalid-training-data-the-output-size-7-of-the-last-layer-does-not-match-the-number-of-classes-1)

[Mohammad Sami]

It's because we set the 10 categories in the setcat_and_table_to_cell.

You should change the variable validcats to the appropriate 7 categories for your case.