"MATLAB was unable to initialize the GPU FFT library" ??

Hi,

 I am using a GeForce GTX 680, 4GB memory GPU (32bit Windows XP, Matlab 2013a) to help my computation. When I am trying to do the 7th time complex to complex ifftn on a 256x256x501 single gpuArray in my code, I got this weird error.

 Strangely, the code passed a 256x256x501 real to complex fftn once and six times 256x256x501 complex to complex ifftn. This problem is repeatable on this array and always happen at the 7th time of ifftn. 

 The error is triggered by a command "temp = ifftn(temp)", where temp is the 256x256x501 single complex gpuArray. I have updated to the latest driver (date: 25/03/2013, Ver:314.22). I use neither C++ kernel nor any mex command in my code. All code is based on the matlab provided functions. 

 Here is the gpuDevice output when the program turned into debug mode because of this error:
                      Name: 'GeForce GTX 680'
                     Index: 1
         ComputeCapability: '3.0'
            SupportsDouble: 1
             DriverVersion: 5
            ToolkitVersion: 5
        MaxThreadsPerBlock: 1024
          MaxShmemPerBlock: 49152
        MaxThreadBlockSize: [1024 1024 64]
               MaxGridSize: [2.147483647000000e+09 65535 65535]
                 SIMDWidth: 32
               TotalMemory: 4.294639616000000e+09
                FreeMemory: 2.592538624000000e+09
       MultiprocessorCount: 8
              ClockRateKHz: 1163000
               ComputeMode: 'Default'
      GPUOverlapsTransfers: 1
    KernelExecutionTimeout: 1
          CanMapHostMemory: 1
           DeviceSupported: 1
            DeviceSelected: 1

By the way, I wonder if this is related to memory issue. Is there anyway to enforce an in-place FFT when using gpuArray on fftn/ifftn? The code randomly runs into memory errors and randomly give one of the below error messages at fftn/ifftn commands when handling large 3D arrays (such as arrays 256*256*something larger than 500): 1. Out of memory on device. MATLAB was unable to allocate sufficient resources on the GPU to complete this operation. 2. MATLAB encountered an unexpected error in evaluation on the GPU.

When this problem happens, the GPU has more than 1.5GB free memory, reported by both gpuDevice in matlab and from a 3rd party GPU monitor program.

The memory problem won't happen on small arrays (e.g. 256*256*256). Executing the code in a fresh started Matlab session reduces the chance of triggering the memory problem (not eliminating it). clear all; and reset() have no noticeable effect on circumventing it.

Answer 1

[James Lebak]

You are correct that the symptoms you mention indicate a memory problem. The GPU FFT requires memory not only for the input and output, but for the weights and intermediate buffers. In general, GPU FFTs require less memory when all dimensions are powers of 2. You might be able to work around the problem by padding the output of the fftn command to a power of 2 with something like

fftn(x, [256 256 512])

The GTX 680 that I have access to has only 2G RAM and is in a Linux machine. On that machine, I can execute an FFT with padding as above, and then repeatedly execute IFFT's of size 256 by 256 by 512 with no issues.

If this workaround does not solve your problem, here are some other questions that might help us diagnose this better:

• What other variables are in memory when you call the IFFT?
• Could you give us a small piece of code that reproduces the problem?
• Is your card a dedicated compute device, or is it also being used to display graphics?

Answer 2

[Max W.K. Law]

You are right. I would add one more observation. While the code jumps into debug mode because of the out-of-memory errors, I can actually do a trick: temp=gather(temp); temp=gpuArray(temp); temp=ifftn(temp); to workaround this problem in the console.

My code makes use of six ifftn outputs to form a 3x3 tensor field, and do a GPU eigenvalue extractions. This process is repeated over 10 times. I guess the problem is a part of memory fragmentation in the GPU. Is there any way to defrag the GPU memory without interfering the main memory which is already full?

I also sometimes run into a situation that temp=gather(temp); will complain a out of memory on the device error. Is there any documentation about the actual things/API being called by Matlab for the commands gather and fftn/ifftn?