MATLAB Answers

Direct GPU-to-GPU Communication with Parallel Computing Toolbox / SPMD

1 view (last 30 days)
Jonathan
Jonathan on 24 Apr 2015
Edited: Jonathan on 30 Apr 2015
I am using spmd to enable parallel computing with multiple GPUs on one workstation. Basically, the GPUs do some calculation, broadcast their results, update their parameters, and iterate. The problem is, using labSend (actually, gplus in my case) to aggregate and broadcast the results is pretty slow. It is first pulling the results off of the GPU, copying to system memory, sending to other workers, then uploading to the other GPUs.
I understand that CUDA now has Peer-to-Peer memory access capability. This way, multiple-GPUs can directly access each other's memory. http://www.nvidia.com/docs/IO/116711/sc11-multi-gpu.pdf This is accomplished with a function like: cudaMemcpyPeerAsync().
Thus, I would like to have a gplus() or labSend() that copies a gpuArray directly to the memory of another GPU on another worker.
Is this possible today? If not, is it something you are working on?
Thanks, Jon

  0 Comments

Sign in to comment.

Answers (1)

Edric Ellis
Edric Ellis on 27 Apr 2015
Edited: Edric Ellis on 27 Apr 2015
Unfortunately, as you observe, Parallel Computing Toolbox currently has no means by which to achieve this. I believe you can use the peer-to-peer memory copying across multiple processes within a single node, which means you could use the GPU MEX interface to copy data.

  1 Comment

Jonathan
Jonathan on 30 Apr 2015
Basically, what I am trying to accomplish is to execute the same function (with different data) on two GPUs simultaneously. Then I sum the results, update some parameters and repeat.
I assume, I could write a MEX function that takes as input a matlab function handle to evaluate, and multiple gpuArrays to operate on. It would return the sum of the results in two different gpuArrays (each array having identical values, but stored on different GPUs), having performed the calculation on multiple GPUs. The problem is, can you have gpuArrays pointing to data stored on different GPUs in a single worker or client?
It seems like the problem here is that, outside of the MEX function, regular matlab scripting cannot handle gpuArrays pointing to different devices, no?

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by