Using CUDA mex files interoperably with gpuArray

I have a 3rd party black box CUDA mex file. Every time I use it, it puts my GPU in a state such that gpuArray cannot subsequently use it. This is in R2020a, but I've observed similar problems in previous versions.

>> a=gpuArray(1)
Error using gpuArray
An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_CONTEXT_IS_DESTROYED

The only remedy once this occurs is to restart Matlab. Is there any way gpuArray and my cuda mex can share use of the same GPU without nuclear devastation?

[Matt J]

If I issue reset(gpuDevice) whenever I switch between usage of the mex and gpuArray, it seems to avoid the CUDA_ERRORs, but this is not ideal, because it clears previous gpuArray variables that I might like to keep around. Is there a less destructive alternative?

[Edric Ellis]

Unfortunately, I suspect there is not. It is definitely intended that you should be able to use CUDA MEX files together with gpuArray. If the CUDA MEX files are putting the GPU into a bad state, then reset(gpuDevice) is indeed the best option. You can use save to save gpuArray data to a file (providing you do this before the GPU is in a bad state).

[Matt J]

It is definitely intended that you should be able to use CUDA MEX files together with gpuArray

Thanks, Edric, but can you speculate what basic mistake the MEX might be making that would put the GPU in a bad state, and how it could be fixed? How exactly is a CUDA MEX file supposed to be aware of and operate within the same CUDA context as gpuArray code?

Answer 1

[Joss Knight]

A third party CUDA MEX file could be built with a different version of the CUDA toolkit, and thus a different version of the CUDA runtime. You will not be able to safely pass gpuArray objects into and out of the function.

To avoid this, obtain the source code and compile the function yourself using MEXCUDA.

That said, there's nothing to guarantee it won't work...but in this case it looks like your MEX function is resetting the device itself and thus destroying the CUDA context (for instance, it could be calling the CUDA C runtime function cudaDeviceReset). You won't be able to use it safely with gpuArray variables in your MATLAB workspace. Again, if you had access to the source code, you could modify this behaviour and even have MATLAB successfully respond to the reset event.

[Matt J]

Thanks, Joss. I do have access tot he source code. What exactly did you mean by "have Matlab successfully respond to the reset event"?

[Joss Knight]

Well, I was being loose with my words. It's possible to have a call to reset in MATLAB successfully clear up persistent state in a MEX function, equivalently it is possible to cleanly reset the device from a MEX function (typically by actually executing the MATLAB function reset(gpuDevice) rather than calling the CUDA API). One of the things you can theoretically do by managing device reset is save GPU variables or return them to the host so the data is not lost.

However, it's pretty rare to need reset the device in a MEX function so perhaps your best bet is to work out why the CUDA context is being destroyed and fixing that problem.

[Matt J]

Getting rid of the cudaDeviceReset's fixed it. Thanks!