Error using nnet.inter​nal.cnngpu​.convolveF​orwardNDBu​iltin. Maximum variable size allowed on the device is exceeded.

I am attempting to utilize the unet3dLayers command to trian a 3D UNet. For the training and validation, I simply ran 64x64x64 blocks that held my data with no errors in the training process. This data was made by combining an imageDatastore and a pixelDatastore. Now that I have a trained network, I am attempting to test the network with the set aside data that I created the blocks in the first place. To my surprise, no matter how I utilize my combinedDatastore, I am left with the same error in the summary, which is "Error using nnet.internal.cnngpu.convolveForwardNDBuiltin. Maximum variable size allowed on the device is exceeded." There are not a ton of resources in the MATLAB documentation as it relates to creating and testing these unet3dLayers networks, especially those that do not first chunk the data into sub-blocks first (See https://www.mathworks.com/help/images/segment-3d-brain-tumor-using-deep-learning.html) . However, due to the fact that I trained the data on this format (64x64x64) I am wondering why the error is occurring in the first place as well as the best way to get around fixing it.

For further context, I did find this forum post, though with the difference in sizes being so large between my data and the one posted below, I am not sure if the same answer is to be recommended. https://www.mathworks.com/matlabcentral/answers/385273-why-do-i-get-maximum-variable-size-allowed-on-the-device-is-exceeded-error-when-running-the-sema

Answer 1

[Matt J]

You can use the memory command to see how close you are to your RAM limit. Basically, though, I would take the error at face value. You are running out of memory.

Why you are not seeing an error during training calls for speculation on our part. We don't know what other variables that you may have introduced into your workspace post-training.