Is it possible to prevent parpool('Processes') to create a large number of open files?
14 visualizaciones (últimos 30 días)
Mostrar comentarios más antiguos
I have noticed the following behavior of the parpool-function in Matlab (2023a). When I use something like
pool = parpool('Processes', 128)
it return with an error
Starting parallel pool (parpool) using the 'Processes' profile ...
Parallel pool using the 'Processes' profile is shutting down.
Error using parpool
Parallel pool failed to start with the following error. For more detailed information, validate the profile 'Processes'
in the Cluster Profile Manager.
Caused by:
Error using parallel.internal.pool.AbstractInteractiveClient>iThrowWithCause
Failed to initialize the interactive session.
Error using parallel.internal.pool.AbstractInteractiveClient>iThrowIfBadParallelJobStatus
The interactive communicating job failed with no message.
This error is due to the fact, that on the used Linux server, the number of files is limited
$ ulimit -n
131072
and it seems that Matlab is opening >1000 files per worker requested for the pool. This seems to be an unreasonable large number of files imho so my question is if I can somehow prevent this?
I know that I can of course increase the limit on the number of files in the OS. I also noticed that
pool = parpool('Threads', 128)
does not suffer from this problem (and the pool has also a much short startup time), but in parfor-loop a piece of Mex-compiled code is used and this is not possible for a pool of threads:
Use of MEX functions is not supported on a thread-based worker.
So an alternative question would be how to use a MEX function on a thread-based worker?
0 comentarios
Respuestas (1)
Paras Gupta
el 14 de Mayo de 2024
Editada: Paras Gupta
el 14 de Mayo de 2024
Hi Stefan,
I understand that you get an error when you start a parallel pool using 'Processes' with 128 workers.
This seems to be a known bug in MATLAB R2023a for cases when the number of workers is more than 100. Please refer to the following bug report for a workaround to resolve the issue:
Also, using a MEX function on a thread based worker seems to be a limitation of MATLAB as of the latest release R2024a. You may contact MathWorks Technical Support for more information on the same.
Hope this answers your query.
0 comentarios
Ver también
Categorías
Más información sobre Parallel Computing Fundamentals en Help Center y File Exchange.
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!