Parallel workers automatically shutting down in the middle of RL parallel training.

Hello,

I am currently training a reinforcement learning PPO agent on a Simulink model with UseParallel=true. The total episodes that it should be running for are 5000 (about 10/11 hours of training), but i'm noticing that as the training goes on, more and more workers of the parallel pool are automatically shutting down, making training slower and slower as it progresses. I start with 8 workers, and they consistently decrease one at a time, until errors are generated.

I've been noticing this consistently in each training that i do, and would like to know if there are any workarounds.

For the parpool, i am letting Matlab start it automatically with all options set to default. I have also tried playing around with the number of workers, but the same thing happens.

[Emmanouil Tzorakoleftherakis]

What errors are you seeing? Maybe try training on a single worker initially to make sure you don't see any errors before moving to parallel.

[Matteo D'Ambrosio]

On a single worker everything works fine, and the errors i get happen after 3000+ training episodes, after the workers have slowly started shutting down one at a time. I've also used the environment validation function.

I've noticed that after training for this number of episodes, i also get the following error (tied to the PPO algorithm):

Dot indexing is not supported for variables of this type.

Error in rl.agent.rlPPOAgent/learnFromAdvantages (line 40)

advantageData.Advantages,this.AdvantageBuffer_,...

Error in rl.train.parallel.AsyncPPOParallelTrainer/processSimOutput_ (line 62)

learnData = learnFromAdvantages(agent,data.AdvantageData);

Error in rl.train.parallel.AsyncParallelTrainer/processFutures_ (line 22)

processSimOutput_(this,out);

Error in rl.train.parallel.AbstractParallelTrainer/run (line 64)

[F,outs,taskIDs] = processFutures_(this,F);

Error in rl.train.TrainingManager/train (line 479)

run(trainer);

Error in rl.train.TrainingManager/run (line 233)

train(this);

Error in rl.agent.AbstractAgent/train (line 136)

trainingResult = run(trainMgr,checkpoint);

Answer 1

[Edric Ellis]

If workers are leaving the pool one at a time while the pool is busy, this almost certainly means that they are crashing. I recommend contacting MathWorks support for help diagnosing and resolving this problem. You could also check by running the following location to see if any "matlab_crash_dump.*" files have been left behind:

jsl = parcluster().JobStorageLocation
dir(fullfile(jsl, '**', 'matlab_crash_dump.*'))

[Matteo D'Ambrosio]

Thanks for the help, i will contact MathWorks support to look into this further.