Error when processing on HPC: Unable to allocate space for the FFT calculation. This might be due to insufficient memory on the GPU.

Hello,

Error message: Unable to allocate space for the FFT calculation. This might be due to insufficient memory on the GPU.

I received this error message when I'm processing multiple images on a Slurm server. The code used both GPU and multi-core computing. The for loop goes over all the images are not parallelized, within each image, the cores work together to produce the result for this simgle image.

The error message shows up after going through around 4000 images. I tried to clear all the variables after completing every single image, and reset the GPU device every 2000 images, and the error message is still there.

The error results in a stop in calculation, the server gets a return 0 message (which means a normal exit on the server).

Please help.

Answer 1

[Joss Knight]

At a guess you are trying to share one GPU between multiple workers on a pool. Depending on how work is scheduled one or two workers may have allocated all GPU memory leaving none for others.

Options:

• Reduce the size of the pool
• If on R2022b or later, try setting the gpuDevice CachePolicy to "minimum"
• Place your code inside a try... catch block and ignore out of memory errors, or use the CPU instead if the GPU errors

[Xiguang Zhang]

Thanks for the suggestion, yes you are correct, I'm trying to share 4 GPUs among 48 workers.

How can I set CachePolicy for gpuDevice? I did not find any documentation about this parameter.

[Joss Knight]

Hmm, good point. I guess that's not out yet, sorry.

Try this:

feature('GpuAllocPoolSizeKb', 0);
feature('GpuFftMaxStoredSizeKb', 0);

But there really isn't any point in sharing your GPUs. They can only do one FFT at a time.

[Xiguang Zhang]

I'm confused now. The parallel part of my code look like this:

parfor i=1:600
    % reconz, phasemap, and fftholo are all gpu arrays
    recfield = ifft2(exp(1i*reconz(i)*phasemap).*fftholo);
    
    something else
end

So are you saying, the ifft2 part is still calculated in series on GPU even though it's inside a parfor loop? Then why I'm seeing an increase in performance when I use parallel computing? Is it because it's faster when doing the other parts?

[Joss Knight]

The ifft is computed in a data-parallel way but there is no overlap between the computations being run on different workers that share a GPU. Some overheads will be reduced but the main gains you see will be the fact that you have 4 GPUs.

[Xiguang Zhang]

Thanks for the information. The error does not show up after reducing the number of workers.