Training Neural Network for Image-to-I​mage-Regre​ssion using Numeric Array

Hi,

I would like to train the Neural Network I created for Image-to-Image-Regression using a Numeric Arrays. I already did it using a combined Imagedatastore, containing the input-data (60x60x1 images) and the expected output-data (60x60x1 images). Now I tried to do the same but with two numeric arrays, one for the input-data (60x60x1xn, n: number of images) and one for the output data (60x60x1xn). First I tried to use the two numeric arrays like this: trainNetwork(input,output,mynetwork,options). The training worked but the results where very different, the netork always predicted the input picture.

Then I created two Array-Datastores, one for the input and one for the outout and combined the two Array-Datastores, like I did it before with the ImageDatastores. I then tried to train my network like this: trainNetwork(combinedDatastore,mynetwork,options) and got this error message: "Invalid training data. For a network with 1 inputs and 1 output, the datastore read function must return a cell array with 2 columns, but it returns a cell array with 1 columns."

I would be really thankful, if someone could help me. Thanks!

[Rahul Gulia]

Could you please direct me to some tips on how did you modify the training and testing data with image as an input and an image as the output. And how did you modify the train model to predict the images from the input dataset (also images).

I am trying to solve a similar kind of problem. I would really appreciate any kind of suggestion on this topic.

Thank you,

Rahul

[Benedikt Hamann]

Hi @Rahul Gulia,

I provided my data to the model in the format of a numeric array with the size image height x image width x color channels x number of samples. I also splitted the data into a training, validation and test dataset, resulting in six arrays (train_in, train_out, val_in, val_out, test_in, test_out). You can also use the imageDatastore format, however, in my opinion it is impractical, espacially when using a remote cluster for the training process. For the training process I used the following options and a output function which stops training prematurely in case the validation perfomance does not increase further.

miniBatchSize = 32;
ValidationFrequency = floor(length(train_in(1,1,1,:))/miniBatchSize);
options = trainingOptions('adam',"InitialLearnRate",0.001, ...
    'MaxEpochs',200, ...
    'Plots','none', ...
    'Verbose',true,'VerboseFrequency',5000, ...
    'ValidationData',{val_in,val_out},'ExecutionEnvironment','multi-gpu',...
    'ValidationFrequency',ValidationFrequency,'MiniBatchSize',miniBatchSize, ...
    'Shuffle','every-epoch','OutputFcn',@(info)OutputFc(info),...
    'OutputNetwork','best-validation-loss');
trainedModel = trainNetwork(train_in,train_out,yourModel,options);
save('trainedModel.mat',"trainedModel")

function stop = OutputFc(info)
N = 5;
stop = false;
persistent bestValRMSE1
persistent valLag 
if info.State == "start"
    bestValRMSE1 = 10000000;
elseif ~isempty(info.ValidationRMSE)
    if info.ValidationRMSE < bestValRMSE1
        valLag = 0;
        bestValRMSE1 = info.ValidationRMSE;
    else
        valLag = valLag + 1;
    end
    if valLag >= N
        stop = true;
    end
end
end

To let the model predict images you have to use a regression output layer and adjust the size of the last convolutional layer to fit the image size (e.g., image height x image width x color channels). I did also not used fully connected layer as it is usually done when the model predicts image sized data.

I can also recomment this guide: Matlab image to image regression

Answer 1

[Sandeep]

Hi Benedikt Hamann,

The trainNetwork function expects the training data to be returned from the datastore's read function in a specific format. For a network with one input and one output, the read function must return a cell array with two columns, where each row contains an input and its corresponding output.

When you tried to train the network using the two numeric arrays, you passed the inputs and outputs as separate arguments to the trainNetwork function. This works for simple cases, but may not be the best approach when dealing with more complex data or when using datastores.

When you created the Array-Datastores for the input and output data, you need to make sure that the read function returns data in the correct format.

Here is an example of a read function definition,

inputDs = arrayDatastore(inputData, 'IterationDimension', 4, 'ReadFcn', @(x) {x, []});

This ReadFcn will return a cell array with two columns, where the first column contains the input data and the second column contains an empty array (since this is the input datastore and there is no corresponding output data).

For more insight on trainNetwork refer the documentation page: trainNetwork Documentation

[Benedikt Hamann]

Thank you!