Large training set in Semantic Segmentation

We are training a SegNet using a dataset composed of 26000 images (and 26000 associated image labels) of 256x256 pixels. The training does not start due to insufficient memory (our GPU has 6.8 GB of available memory according to the gpuDevice() command) even if the MiniBatchSize is set to 1. Obviously, we are using a pixelLabelImageDatastore to pass the dataset to the training function. It seems that the only way to solve this problem is to reduce the amount of the images of our dataset (but we would need even MORE).

We would like to know what is the best way to overcome this problem (splitting the dataset into subsets, using DispatchInBackground to pass fewer data to the GPU)?

Thank you in advance.

[Joss Knight]

Your data exists as a single huge 4-D array, or as files?

[Gaia Pavoni]

As files. One file per image.

[Joss Knight]

Can you give some example code? Simply calling trainNetwork with an imageDatastore does not attempt to load all the images onto the GPU; the amount of memory needed is related to the size and depth of the network and the MiniBatchSize. So if you are running out of memory with a MiniBatchSize of 1, you should run out of memory with a dataset of a single image, and reducing the number of images will make no difference. Either you would need a bigger GPU, or a different network design that doesn't need as much memory.

[Birju Patel]

Can you paste in the error message so we can see where the error occurs while running the example?

Answer 1

[Gaia Pavoni]

Thanks a lot for your answer, the training process correctly starts using a dataset of 16000 images with a MiniBatchSize of 16. Our GPU has 8GB of RAM, we are using Segnet exactly as illustrated in the Camvid example using adam as optimizer.

[Marya Rasib]

Can you share your code with us please, i am facing the issue, " {"out of memory. Type "help memory" for your options. caused by: out of memory. Type "help memory " for your options"}. Thanks in advance