Communicating between tasks and submitting client in parallel jobs

2 visualizaciones (últimos 30 días)
emarch
emarch el 10 de Dic. de 2018
Respondida: Prashanth Ramesh el 14 de Dic. de 2018
I've got a long job I've broken into several tasks to run on MS HPC and I am trying to figure out a way to send progress back to the submitting client computer. I'm using createJob/createTask to run the job. Right now I'm polling for the individual task states and this works, but I'd like to get some insight into where each task is in it's progression (e.g., 60% complete). Some ideas I'm condsidering or have tried include:
Using the task objects UserData property, but this appears to only be updated on the matlab worker (for example, one can print it to the Diary just fine from within the function the task is running).
The DataQueue object looks promising but it looks like it's only available when using parallel pools.
I've thought of writing to the diary and then on the calling/client machine having something read these every couple of minutes across the network and present a progress bar for each task. My principle hesitation being this seems a little kludgy.
Any other ideas or suggestions?
Thanks

Respuestas (1)

Prashanth Ramesh
Prashanth Ramesh el 14 de Dic. de 2018
While I admit there does not exist a clean documented way of achieving your workflow, I believe implementing parallel pool with a single worker may be a workaround for this issue. Please note that MATLAB
does not add additional overheads between CPU and parallel pool, if the worker size is 1. The only caveat however is that your existing code for the jobFunction should be compatible with "parfeval" .
parpool(1); % Only 1 worker required
Q= parallel.pool.DataQueue;
Q.afterEach(@updatWaitBar);
future = parfeval(@jobFunction, 0, queue);
You could also consider using a similar workflow as mentioned in the below link, although it is implemented for "parfor", the same can be achieved for jobs:

Categorías

Más información sobre Parallel Computing Fundamentals en Help Center y File Exchange.

Etiquetas

Productos


Versión

R2018b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by