I want my 200 jobs to wait in a queue so that they occupy only 40 workers out of 100 workers in our cluster

We have a cluster with 100 workers in our lab, and I am allowed to use up to 40 workers. The problem is that I cannot submit more than 40 jobs at once, because those extra jobs will use the idle workers of my colleagues. How can I tell MATLAB job scheduler to limit my access to only 40 workers? I don’t think the solution is setting “NumWorkersRange” to [1 40].

Answer 1

[Edric Ellis]

If you're using an MJS cluster, then NumWorkersRange on a single independent job is the way to do this. You'll need multiple createTask calls, a bit like this:

c = parcluster(...);
j = createJob(c, 'NumWorkersRange', [1 40]);
for idx = 1:N
  createTask(j, @myFcn, numOutputArgs, {inputArgs, ...});
end
submit(j);

[Data Analysis]

Prompt and accurate reply!