Action of the RL agents actions change when deployed in a different enviornment

Hi all,

I have an RL agent trained in a environment (env 1- a simulink model). The sample time of the agent is 0.1s. It uses a variable step solver (DormanPrince) to solve each episode. After training is complete I export the same agent to a different enviornemnt (env 2 - a more complex environement compared to env 1) and deploy without changing any parameters. This environement does not have any randomness built into it. Also it is solved suing a variable step size solver (DormanPrince). However, when I run simulations with env 2 with the same initial condtions, I get different results. (Les say the trajectory I am calculating changes each time I run the simulation).

Why does this happen even when there is no randomness in the simulation model? If I run the simulation without the agent with same initial conditions I get a solution which does not change everytime I run it.

Plese let me know if anyone has encountered this. Thank you!!

Answer 1

[Emmanouil Tzorakoleftherakis]

A couple of suggestions/comments:

1) You mentioned env1 and env2 are different - why are you expecting to see the same results? Variable step solvers can lead to different numerical results if the stiffness of the underlying equations that are integrated changes. Even if env1 and env2 are modeling the same system, the more complex version is likely to be more stiff, which can change the simulation results.

2) Which agent are you using? Some agents are stochastic by nature, so unless you fix the random see generator, you will see different results

[Jayalath Achchige Damsara Udan Jayarathne]

Hi,

Thank you very much for the reply.

1) Yes, I agree with that. But I observed that if I run the simulation in environment 2 without the RL agent, the result I get is almot identical. (at least similar to e-04). But if I run the simulation in with the RL agent, there is a considerable difference in the result. (of course it will be different because of the actions of the RL agent. but if I run two simulations with the same initial conditions with the RL agent, the end results (I am measuring a error state which needs to be < 0.005) I get are not the same. I am confused about this discrepency.

*I am NOT expeting the same result from env 1 and env 2, but I am comparing results in env 2 for same initial conditions (without any randomness).

2) I am using DDPG which is a deterministic agent so I dont think the action chosen by the agent is stochastic at all.

[Emmanouil Tzorakoleftherakis]

If I understand correctly, running two simulations on the env2 environment, both using DDPG agent and same initial conditions leads to different results? Interesting. Have you tried logging some signals, including actions and observations, and comparing them on Simulation Data Inspector to see where the discrepancy begins?

[Jayalath Achchige Damsara Udan Jayarathne]

Correct. Yes, I looked at the actions and observations. The are not drastically different, but certainly different. Does this has anyhting to do with the sample time of the agent?

[Emmanouil Tzorakoleftherakis]

Don't think so, the agent will generate actions at fixed time intervals, every Ts seconds. Do you have any other components that may be nondeterministic in your model? Are you simulating using the Play button or using the 'sim' command from Reinforcement Learning Toolbox?

[Jayalath Achchige Damsara Udan Jayarathne]

No I checked my model several times, it does not have any other stochastic components. I used the play button.

[Emmanouil Tzorakoleftherakis]

Don't have any other ideas, maybe technical support can help if you can share a reproduction model.

[Jayalath Achchige Damsara Udan Jayarathne]

Yes, that's what I'm going to do next. Thanks for your suggestions!