Agent is suddently doing random actions and training diverge

Hello,

I am training an DQN agent to replace a controller. Everytime the agent is about to converge, it starts to make random move and diverge. Although greedy epsilon was set to (0.3, 0.5, 0.6 etc.). Any idea what the reason might be?

Thanks in advance!

Answer 1

[Emmanouil Tzorakoleftherakis]

This is normal behavior - one common misconception is that once the reward starts going up, it will remain up. This is not true as the agent may start exploring a completely different part of the state space and that can lead to sudden dips in the reward as you can see.

Once you observe good behavior for a few episodes in a row/good average behavior over a number of episodes, that's a good indication that you can stop training. So I would stop training after episode 50/60 in your case and see if the result works, or I would let it train for more time and see if it recovers/surpasses the previous max (this is also common)