How to handle invalid/illegal actions/moves in Reinforcement Learning toolbox?

I have implemented a Reinforcement Learning Algorithm, where some actions are not legal/valid depending on the state. What is the best way to deal with this problem? I have tried to give negative rewards and stop the episode if an illegal action is selected, but it does not seem to work. I also tried to just ignore these actions, but then the agent keeps making the same move until it reaches the max episodes during the training. My best approach was to make the next possible action, instead of the selected one and although it has better results than the other options, it does not seem to be a good way to deal with it. I have read on the AlphaGo paper that "Illegal moves are masked out by setting their probabilities to zero, and re-normalising the probabilities for remaining moves." Is there a way to implement this on Matlab and if not what would be the best way to avoid the invalid moves ?

Answer 1

[Rajani Mishra]

To implement your own custom reinforcement learning algorithms, you can create a custom agent.

Specify properties of the agent for creating and training the agent. Refer this link for creating custom agents : https://www.mathworks.com/help/reinforcement-learning/ug/custom-agents.html

I found one example of a Reinforcement learning agent, link for which is:https://www.mathworks.com/help/reinforcement-learning/ug/train-q-learning-agent-to-solve-basic-grid-world.html