DCS parallel config validation - make the scratch directory during validation otherwise validation fails (any workarounds)

DCS 128 node license
Parallel configuration - Parallel Submit function in the configuration manager is parallelSubmitFcn- {@parallelSubmitFcn, 'cluster.*******.***.edu', '/lustre/scratch/USERID/matlab'}

However If the folder /lustre/scratch/USERID/matlab does not exist, validation of the parallel configuration fails. Is there anyway to create the folder using 'mkdir' just before the validation runs.

once it is made it can always be used by Matlab.

Right now we ask Users to ssh and create the folder but was wondering if the there was way to automate it before validation as 'validation' is the first step users perform before using MATLAB on our cluster.

P.S.- Have looked into the Submit scripts tailored for our cluster parallelSubmitFcn.m , distributedSubmitFcn.m, findResource.m and the validateConfig.m in the distcomp toolbox but could not find a place to make the directory.

[Konrad Malkowski]

Have you considered creating a global remote data location for all users to use?

[Thomas]

Just to clarify would the 'remote data location' be to store the Parallel submit function (parallelSubmitFcn) and point all users to it?

[Konrad Malkowski]

No,

the remote data location is directory where you store job information on workers, for example: /lustre/scratch/USERID/matlab

What you might want to consider is creating a global directory for all MATLAB jobs.

[Thomas]

How would user permissions affect the data in this directory? We do not want users seeing data form other users here..

[Konrad Malkowski]

The directory would behave just like any other unix directory. By default user files would have their default permission settings, so if a user is has default settings of chmod 700 then he/she will be the only one able to do anything with the files.

Under remote and non-shared submission schemes, users are required to login to cluster headnode in order to submit the jobs.

In the shared submission scheme, they are presumably already writing the files with correct permissions.

[Thomas]

Konrad,

We do not want to create the directory for all users on our cluster, but only a small fraction of those users that use MATLAB (less than 2%) and hence we did not want to go ahead and put it in for all our numerous users and clutter their scratch space with an extra folder. My question was can we create this space while validation of the MATLAB config from the desktop since that is one of the first steps we ask our users to perform after configuring MATLAB with our cluster.

Validation fails if that directory does not exist.

Can we create the directory during the 'findResource' part of the validation?

[Konrad Malkowski]

Hi Thomas,

No, at the moment there is no way of doing this automatically during validation.

I will add it as an enhancement request for a future release.

[Thomas]

Thanks, Konrad!

Answer 1

[Thomas]

This request has been added as an enhancement form a future release..