How to train a network and reconstructing the dataset on the fly

Hello there!

I want to train a pretrained CNN, using image data that i create myself (based on saved data), because of the amount, i want to create the images on-the-fly, while runing the the training. and i'm not sure how to read and use

i.e, i have a few folders containing >1600 *.mat files of amplitude and phases of my image. but the dataset i want to train on is a calculation based on those *.mat files. using the calculations, the dataset can grow even uo to a million images.

since i can't save million images, how can i create smaller datasets and train/validate the network in one go?

do someone has an example of using mini-batches etc.?

thanks a lot!

Answer 1

[Sanjana]

Hi,

I understand that you want to train the network using mini-batches and looking for a way to manipulate dataset on the fly while training.

One possible solution is to use custom training loop.

Please refer to the following link for a reference to write custom training loop in MATLAB,

https://www.mathworks.com/help/deeplearning/ug/train-network-using-custom-training-loop.html

Please refer to the following example, for understanding manipulating data within a training loop and usage of mini batches for training,

https://www.mathworks.com/matlabcentral/fileexchange/76860-human-pose-estimation-with-deep-learning

Hope this helps!