Mini batch size for SeriesNetwork

Hi!

I have got some issue, it seems that miniBatchSize does not divide my training data into batches, whole matrix of 2e6x15 goes though training per one iteration. I can see in options function that miniBatchSize gets a value, tried everything from 2 to 1M. Thank you for your answers :)

[Maria Duarte Rosa]

Hi Maxim, can you provide more details on what made you conclude that the whole matrix 2e6x15 goes through training per one iteration?

[Maxim Abdullayev]

For example in the gui wondow during training process, iterations per epoch is equall to 1, and it really is equal in the graph and so on. I dont have possibility to post screenshot of training process, but here is the code. I have tried differen parameter values, but nothing works.

size  = 50000;
layers = [  sequenceInputLayer(15)
%    sequenceFoldingLayer
%     lstmLayer(2,'OutputMode','sequence')
            fullyConnectedLayer(100)
            dropoutLayer(0.5)
            fullyConnectedLayer(1)
   % sequenceUnfoldingLayer('unfold')
            regressionLayer];
validationFrequency = floor(numel(trainOUT)/size);
options = trainingOptions('adam', ... % sgdm rmsprop adam
    'MaxEpochs',50, ...
    'InitialLearnRate',1e-2, ...
    'LearnRateSchedule','piecewise', ...
    'LearnRateDropFactor',0.1, ...
    'LearnRateDropPeriod',20, ...
    'Shuffle','never', ...
    'MiniBatchSize',size, ...
    'ValidationData',{testIN',testOUT'}, ...
    'ValidationFrequency',validationFrequency, ...
    'ValidationPatience',5,...
    'Plots','training-progress', ...
    'Verbose',false);
net22 = trainNetwork(trainIN',trainOUT',layers,options);

Answer 1

[Maria Duarte Rosa]

Hi Maxim,

Thanks for providing more details.

If your data is in a D x S matrix format (D being 2e6 and S being 15) MATLAB assumes that this is a single observation problem with 15 time-series each being 2e6 points long. For each epoch, we have only 1 iteration and so the mini-batch size option is ignored because it doesn't apply to just 1 observation.

If you'd like to break the time-series into smaller chuncks of data that are treated as different observations, you can do that using the sequenceLength parameter in trainingOptions, by providing a positive integer with the desired sequence length:

https://uk.mathworks.com/help/deeplearning/ref/trainingoptions.html#bu59f0q_sep_mw_284697ea-2219-4c2f-a036-3024e6f2f313

Depending on how many smaller sequences this generates then the mini-batch size parameter can be used to control de size of the mini-batches as you would expect.

If you'd like to input more than one observation (each observation being D x S, i.e. D data points and S time-series) then it's better to use N x 1 cell arrays (each cell containing a D x S matrix), where N is the number of observations. Please see here for more information:

https://uk.mathworks.com/help/deeplearning/ref/trainnetwork.html#mw_abbda1a5-374e-49fe-ad55-d9191146ecea

I hope this helps.