How do I deal with super larger image in semantic segmentation?

I used to use Keras to train and predict on images larger than 4,000 * 4,000 px2. Due to GPU memory limits, I cannot directly train on that large images (but can do prediction at that size).

In Keras, the user can set the input size of the input layer to [None, None, 3] for fully convolutional models such as UNet, so that I can easily train on patches (1,024 * 1,024 px2) and perform predictions on original images using the same model.

However, I did not find a way to do this in MATLAB. Is it possible to train and predict on images of different sizes using the same model? Or maybe an alternative method to deal with super large images?

BTW, to crop original images into patches and to stich them back after prediction seem not to be the best solution to me due to potential edge effects.

Answer 1

[Maneet Kaur Bagga]

Hi 云帆,

As per my understanding to achieve training and prediction on images of different sizes using the same model by employing "sliding window inference" or "tiling". This method involves breaking down large images into smaller overlapping patches, making predictions on these patches and then stiching them back together to form the final prediction. This helps migitate the memory limitations imposed by larger images.

To address potential edge effects when using the sliding window approach, padding around the edges of the image patches can be applied to ensure that the model recieves context from neighbouring regions.

Please refer to the following MATLAB documentation for better understanding:

sliding window : https://www.mathworks.com/help/dsp/ug/sliding-window-method-and-exponential-weighting-method.html

Hope this helps!

Answer 2

[Shreeya]

Hello,

I understand you want to train large sized images on a neural network in MATLAB. The “inputLayer” function in MATLAB can be used for the same. For networks that support variable sizes for the batch or time dimensions, you can specify the size of the corresponding dimension as “NaN”. Refer to the documentation linked below to understand more about this function:

https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.inputlayer.html

Alternatively, you can also try down sampling the images before training. The MATLAB answer linked below details about this:

https://www.mathworks.com/matlabcentral/answers/55997-how-to-downsample-image-in-both-horizontal-and-vertical-direction

Hope this helps!