Differences createCommunicatingJob( ..., 'Type','pool') and ...'Type', 'SPMD')

I'm just introducing myself into Parallel Computing Toolbox and trying to organize all its functionality - for better understanding purposes.

Now, I'm wondering:

• createJob(...) creates an Independent Job (check!)
• createCommunicatingJob(..., 'Type', 'SPMD') creates a communcating Job which runs simulatneously on all workers, due to synchronized interprocess communication purposes, bla, bla... (check!)

BUT:

• createCommunicatingJob(..., 'Type', 'Pool') creates a communicating Job which: A 'pool' job runs the specified task function with a MATLAB pool available to run the body of parfor loops or spmd blocks.

WHAT?!?:

• the inside of a parfor loop? parfor is independent so, why classifying as CommunicatingJob? (hmm#1)
• spmd tasky must be run simultaneously .. otherwise communication might be quite tricky ... (hmm#2)

So, where's my understanding problem ...? I'm a little ... confused ...

Enlightenment whished-for ... :)

Cheers, Tobi

Answer 1

[Thomas Ibbotson]

Both spmd and parfor require a matlabpool to be open to send their work to each worker in the pool. You can create the matlabpool on your client using the 'matlabpool' function, which creates an interactive matlabpool tied to the lifetime of your MATLAB session.

If however you wish to run your code in the background, or submit it to a cluster to run later, you create a communicating job of type 'Pool'. This creates a job that starts a matlabpool on the cluster, using one of the workers as the 'client' and the remaining workers to form the matlabpool.

[Tobias]

Hi Thomas,

thanks for the quick response! But unfortunately it does not resolve my understanding problem.

If however you wish to run your code in the background, or submit it to a cluster to run later, you create a communicating job of type 'Pool'. This creates a job that starts a matlabpool on the cluster, using one of the workers as the 'client' and the remaining workers to form the matlabpool.

This is quite exactly what "batch" is doing ...

BUT my question focuses on WHY createCommunicatingJob(...,'Type','Pool') is declared as a Communicating JOb IF you can use parfor which is an Independet job with no communication!

I understand a Communicating Job as a Job WITH interprocess communication (labSend, labReceive ...). So why (just for my understanding) can you start create a non-communicating job with an createCommunicatingJob-Contructor ... From my point of view - implying that I am understanding everything right (which I want to find out here) - it is badly overloaded!

I hope you understand my problem. I do need to explain it to some students - so I have to understand it in a very detailed way!

Cheers, Tobi

[Thomas Ibbotson]

Hi Tobi,

I agree that no communication occurs during the code within the body of the parfor loop. However, using parfor requires a matlabpool and in general matlabpools have communication between workers (when using 'spmd' with labSend, labReceive or with distributed arrays). So createCommunicatingJob(..., 'Type', 'Pool') allows you to write code with or without communication between workers, but given that there is potential for communication it is referred to as a communicating job.

I think it's important to reiterate the point that the important thing about a communicating job of type 'Pool', is that it starts a matlabpool, using one of the workers as the 'client'.

Note that this is not exactly equivalent to 'batch', as 'batch' does not automatically start a matlabpool for you on the workers, unless you specify the 'Matlabpool' parameter.

I hope I have understood your problem correctly and that this helps, let me know if there's anything else I can help with.

Cheers, Tom

[Tobias]

Hi Tom,

sorry, I did not know that I do not get e-mail-notified if you answer my comment ... anyways, thank's again for enlightment ;) and I think I got it.

Let me just summarize it:

'batch'

• is of type simplejob
• requires matlabpool -> starting explicitly (via 'batch' or before)
• starts a worker as the client
• implicit decision whether independent or communicating via spmd and parfor statements

createJob()

• is of type Job:Independent
• matlabpool is implicit -> scheduler
• no worker as client !? (since embarassing parallelity and thus autonous, independent task processing)
• mutliple independent tasks can be defined, which (can) run asynchronously
• no parfor oder spmd statement allowed

createCommunicatingJob

• is of type Job:Communicating
• CAN (not must be) Communicating
• matlabpool is implicit -> scheduler
• if Type 'SPMD' ... inner code of spmd block (labsend /receive) but no explicit smpd statement itself AND no extra worker as the client
• if Type 'Pool' ... inner of parfor or spmd AND extra worker
• just ONE task is started which is running synchronously on all workers

Might that all be correct?

Cheers, Tobi

[Thomas Ibbotson]

Hi Tobi,

I think there is some confusion over the role of the 'matlabpool'. From the matlabpool documentation:

matlabpool enables the full functionality of the parallel language features (parfor and spmd) in MATLAB by creating a special job on a pool of workers, and connecting the pool to the MATLAB client.

This means a matlabpool is only required if you have 'parfor' or 'spmd' in the code you wish to run.

'batch' however allows you to run any code on a cluster, regardless of whether it contains parallel language features or not. Hence, a matlabpool is not in general required for 'batch'.

Similarly for createJob() and createCommunicatingJob(..., 'Type', 'spmd'), these allow you to run any code on the cluster and therefore do not require a matlabpool.

Apart from that misconception about 'matlabpool', which I hope I've clarified here, I believe all your other statements are correct.

Thanks, Tom

[Tobias]

enlightment proccess ... completed! :D

Thanks a lot for your patience and your explaining skills ... ;) This helped a lot. Next time I will double(-cross) iterate the manual following "RTFM" ;)

cheers, tobi

[Thomas Ibbotson]

Hi Tobi,

I'm glad that helped. I don't think that this is a particularly easy set of concepts to grasp, it took me a while to fully understand it all. Although all the information is contained in the manual, it often takes someone else who understands it to help link it all together. I certainly didn't learn it by just reading the documentation, I had lots of discussions with my colleagues, and I'm happy to be able to pass on what I have learnt.

Thanks, Tom