Open parallel pool within workers / Hierarchichal parallel runs
15 visualizaciones (últimos 30 días)
Mostrar comentarios más antiguos
Sebastian Sas Brunser
el 26 de Jul. de 2022
Editada: Sebastian Sas Brunser
el 28 de Jul. de 2022
Good day,
I have a numerical simulation that runs in parallel with multiple workers. Now I'm trying to optimize the input parameters by running this simulation with the genetic algorithm (ga) tool. So far, so good. This works.
The thing is that if I try to run the genetic algorithm in parallel using:
options = optimoptions('ga','UseParallel',true);
there's the following error:
A parallel pool cannot be started from a worker, only from a client MATLAB.
If I comment out the parpool call inside the objective function (in the simulation) I see that the ga is effectively running multiple times, but I can also see that the objective function is running only in one worker/core.
Is it possible to assign workers to the ga (for example 5) and then assign 8 workers to each ga run? (meaning a total of 5x8=40 cores?) . So this will have 5 ga runs in parallel with each run using 8 workers to solve the numerical simulation.
Thank you for your help in advance!
Best
Sebastian
0 comentarios
Respuesta aceptada
Matt J
el 26 de Jul. de 2022
Editada: Matt J
el 27 de Jul. de 2022
So, first of all, you shouldn't be calling parpool inside your fitness function. That should happen before the optimization starts.
You cannot open hierarchical pools, but if you set UseVectorized=true (instead of UseParallel),
options = optimoptions('ga','UseVectorized',true);
then you can split both the population members and other parallel tasks performed on them across a single pool inside your fitness function. It could like the following, as an example:
parpool(40)
x=ga(@fitnessFcn,...,options)
function f=fitnessFcn(X)
numTasks=8;
numPop=size(X,1);
[I,J]=ndgrid(1:numPop,1:numTasks);
contribution=zeros(size(I));
parfor n=1:numel(I)
i=I(n); %index of i-th popoulation member
j=J(n); %index of j-th task
x=X(i,:); %i-th population member
contribution(n)=fitnessSubfunction(x,j); %fitness value of combination (i,j)
end
f=sum(contribution,2);
end
7 comentarios
Walter Roberson
el 28 de Jul. de 2022
If the output for one population member depends upon which group it is in, or upon what has already been calculated for other population members, then ga() is not a suitable program. ga() depends upon the calculations for population members being independent. The output for one population member must be decoupled from the output for any other population member. If ga() re-evaluates a population member later, it must get the same result it got before (so no randomness either.)
UseVectorized would be for cases where you were somehow able to do the ray tracing for several population members in parallel. Those "massive amounts of simple math calculations" can potentially be done with respect to more than one population member at a time.
But that depends on what it means to mutate the parameters. Given two different population members, is the same amount of work done for the calculations ? For example if one of the parameters controls the "size" of an object, do you keep the same relative calculation resolution anyhow, or is the calculation resolution absolute? Like size 2 vs size 3, fixed calculation resolution 0.01 so array size 200 in the first case, 300 in the second case? Or fixed at (say) 100 points in the dimension, so absolute size 2/100 in the one case, absolute size 3/100 in the other case?
If the different parameters are used to generate a 3D array, and the array is always the same size, then perhaps you can make it a 4d array, one 3D slice for each population member ? If, though, the 3d arrays are not always the same size for different population members, then vectorization is not going to work.
Más respuestas (0)
Ver también
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!