Debug mode for RL agent networks

Is there any way to enter debug mode, to see what is happening inside the RL agent nets while training is running?

Some of my layers output NaN and I would like to know which one. I would also like to monitor to outputs of each layer.

Thanks in advance!

Nicolas

[Nicolas CRETIN]

It would maybe be a solution to add output layers to the network and display their result, but it's tedious.

Answer 1

[Shantanu Dixit]

Hi Nicolas,

It is my understanding that you want to monitor the outputs of each layer and debug the RL agent nets while the training is running.

Similar to analyzing deep learning networks you can call the forward method for each layer of the agent's network to analyze the corresponding output during the training.

[Y1,...,YK] = forward(___,'Outputs',layerNames)

here 'layerNames' correspond to a string array, with 'layerNames(k)' representing kth layer of the agent's network, the corresponding outputs are stored in 'Yk'

Briefly you can follow the below steps:

• Extract layer names from the actor network into a string array 'layerNames'
• Convert Observation Buffer to dlarray for processing into the network
• Forward Pass Through Each Layer using the forward method.

Refer the below code for monitoring the layer outputs corresponding to one observation after updating the actor

%% actorNetwork refers to the DNN of the actor
%% for one observation
dlX = dlarray(observationBuffer(:,:,1), 'CB'); %% format in which the network takes the input
[Y1, Y2, Y3, Y4, Y5, Y6] = forward(actorNetwork, dlX, 'Outputs', layerNames);

layerOutputs = {Y1, Y2, Y3, Y4, Y5, Y6};
for i = 1:numel(layerOutputs)
    % disp(['Layer ', layerNames(i), ' output:']);
    if any(isnan(extractdata(layerOutputs{i})), 'all')
        disp(['Layer ', num2str(i), ' output contains NaNs']);  %% check for NaNs
    end
end

For a better understanding on forward pass and the custom training loop procedure, refer to the following MathWorks documentation

• www.mathworks.com/help/deeplearning/ref/dlnetwork.forward.html
• www.mathworks.com/help/reinforcement-learning/ug/train-reinforcement-learning-policy-using-custom-training.html

[Nicolas CRETIN]

Thanks a lot Shantanu Dixit

This is exactly what I want to do !