How to close client node when running a parpool job across multiple nodes of a cluster using slurm scheduler integration

I have access to a cluster of nodes. Each node has 64 physical cores. My goal is to run a job in parallel across 2 nodes (128 cores). To do this I create a remote cluster profile that utilises direct slurm integration. I first submit a job script to slurm:

#!/bin/bash

#SBATCH -A account

#SBATCH -p partition

#SBATCH -t 12:00:00

#SBATCH -J job

module purge

module load rhel7/default-peta4 matlab

matlab -nosplash -nodisplay -nodesktop -r "EmitCue(${1},${2},'${3}'); quit" # > matoutfile

which calls a matlab script (EmitCue.m). This matlab script requests the resources (a pool of workers):

clusterKNL = parallel.cluster.Slurm;

clusterKNL.NumWorkers = 128;

clusterKNL.NumThreads = 1;

clusterKNL.JobStorageLocation = JobStorageLocation;

clusterKNL.SubmitArguments = '--time=12:00:00 -A account -p partition -N 2 --ntasks-per-node=64 --cpus-per-task=1 -J job'

parpool(clusterKNL,128)

and then goes on to performs the desired computations.

The call to parpool in EmitCue.m itself submits a job to Slurm, which results in the required resources being allocated (in my case 2 nodes with 64 cores each, giving 128 nodes). Note, however, that there are now 2 jobs running simultaneously - the first job (job 1) that was used to open the matlab script EmitCue.m, and the second job (job 2) that was used inside EmitCue.m to request acess to 2 nodes. Now here is my problem. It is only possible for me to request entire nodes on the cluster; I can't request a subset of cores on a node. Hence, when I run the above code I end up with an entire node (I think this is called the 'client') of 64 cores being used just to request 2 more nodes and delegate the job across the 128 workers on those nodes (job 2). None of the cores on the client contribute to the worker pool. This is hugely wasteful. I have limited computing credits that are measured in terms of hours spent on a node so I want to avoid this.

Thanks in advance

Answer 1

[Edric Ellis]

You would be better off if you can submitting only a single job to the cluster, have that job be a batch job with the Pool option specified.

This means you need to set up the cluster profile so that it works from your client, at which point you can do this:

clus = parcluster('MyRemoteSlurmCluster');
job = batch(clus, @performComputations, 1, {inputArgs}, 'Pool', 127);

Where performComputations is your function that uses parfor or whatever. Specifying 127 for the Pool parameter means that the cluster will schedule a total of 128 worker processes - one for the leading worker, and 127 for the parallel pool attached to that leading worker.

[James Heald]

Hi Edric.

Thanks for your response.

"You would be better off if you can submitting only a single job to the cluster" - Are you able to suggest how I can do this? The problem I am faced with is that a pre-condition to doing what you're suggesting is that I need to submit a job to slurm to first gain access to a (client) node, at which point I am in the same situation as outlined in my original post. This approach is described here (https://docs.rc.fas.harvard.edu/kb/parallel-matlab-pct-dcs/) under 'Submitting DCS jobs directly through SLURM'. But maybe I am mis-understanding what you are suggesting?

[Edric Ellis]

The approach I suggested requires a cluster configuration that can submit to the cluster directly from your desktop machine, rather than submitting a job solely to submit a further job.

However, the approach in "Submitting DCS jobs directly through SLURM" looks like it should work - note the "exit" following the "batch" command - your "launcher" job doesn't need to wait for the results to complete, you can go back in and collect those later.

[James Heald]

Hi Edric,

Can I just check that the batch approach you suggest will work for a communicating job of the kind:

function performComputations

for outer_iteration = 1:1e3 % outer loop

    f = zeros(1,1e2);
    parfor inner_iteration = 1:1e2 % inner loop
        f(inner_iteration) = do_something(randn); % perform computation based on stochastic input
    end
    g = do_something_else(f); % perform computation based on all results of the inner loop

where each iteration of the inner loop can be performed independently, but the outer loop cannot proceed until all iterations of the inner loop have been performed. I ask because I notice that the batch command opens 127 separate Matlab sessions (I'm used to using parpool where only 1 session is open). Also, the example mywave function shown in the documentation here (https://uk.mathworks.com/help/parallel-computing/run-a-batch-job.html) seems to be an independent job where each worker can run independent of the others to solve mywave:

function mywave

for i = 1:1024
    A(i) = sin(i*2*pi/1024);
end

[James Heald]

Any idea how I can view the command window output as the job is running? The slurm output file that I specified in SubmitArguments of the cluster profile (i.e., -o filename.out) only shows each of the 128 Matlab sessions being opened (e.g.,

...

[95] < M A T L A B (R) >

[95] Copyright 1984-2019 The MathWorks, Inc.

[95] R2019b Update 1 (9.7.0.1216025) 64-bit (glnxa64)

[95] September 26, 2019

...

[24] To get started, type doc.

[24] For product information, visit www.mathworks.com.

...

[Edric Ellis]

To your first question about parfor and workers - the batch job with a 'Pool' parameter is equivalent to launching a client MATLAB somewhere and having it call parpool for you before executing the task function.

parpool does always launch additional processes - it's just that using the batch approach, you can see all those processes in the top-level job that gets submitted to the scheduler.

There are two batch examples on the page you refer to - the second one https://www.mathworks.com/help/parallel-computing/run-a-batch-job.html#bu62o45 is what you're trying to achieve - the function that you're submitting using batch uses a parfor loop internally which runs on the parallel pool automatically created for you.

Getting the command-window output while the job is running can be tricky - you can check the Diary property of the batch job object - but if you can't access that from your client machine, I'm not really sure what to suggest.

[James Heald]

Thank you Edric for all your help and education.

I did try calling Diary in the matlab function called by the launcher job but it didn't work. I think the problem is that we exit this matlab function after calling batch (in order to close the client to solve my original problem). From what you're saying it sounds like that if you close the client (as I now do), it's not possible to get the command-window output while the job is running. If this is so, can i ask how I can access the command-window output once the job is finished?

[Edric Ellis]

The Diary property should work, but under some circumstances (unfortunately it's quite dependent on your cluster setup), you can only get to it once the job has finished. (I presume you are able to get to the job object in order to fetch the output arguments etc.?)

[James Heald]

I'm not sure I understand where I should be calling Diary and accessing the job object. I have 2 matlab scripts. Script 1: I call batch and then exit. Script 2 is performComputations.m (which batch itself calls) which does the actual work. I assume I should call 'diary(job_object)' in script 1 after calling batch but before exit? Should I be trying to access the job object in script 2 (performComputations.m) at the very end of the script to see the command-window output? Is Diary a field of fetchOutputs that I can save to a file to view offline?

If I just call 'diary(job_object,filename)'in script 1 after calling batch but before exit, should I find a file called filename with the command-window output once the job has finished?

[Edric Ellis]

Once your batch job is complete, you somehow need to get back to the job object it created. If your desktop machine can see the appropriate JobStorageLocation, then that would be sufficient. For example, you might need to do something a bit like this: https://www.mathworks.com/help/parallel-computing/examples/run-batch-job-and-access-files-from-workers.html#RunBatchJobAndAccessFilesFromWorkersExample-4 to find the job that you submitted. From there, you can do:

job = findJob(clus, 'ID', 3); % or whatever
outputs = fetchOutputs(job); % get the output arguments of the function
diary(job) % display the command-window output.