How can I extract a trained RL Agent's network's weights and biases?

How can I extract a trained RL Agent's network's weights and biases?

My network is:

statePath = [
    imageInputLayer([numObservations 1 1], 'Normalization', 'none', 'Name', 'state')
    fullyConnectedLayer(NumNeuron, 'Name', 'CriticStateFC1')
    reluLayer('Name', 'CriticRelu1')
    fullyConnectedLayer(NumNeuron, 'Name', 'CriticStateFC2')];


actionPath = [
    imageInputLayer([1 1 1], 'Normalization', 'none', 'Name', 'action')
    fullyConnectedLayer(NumNeuron, 'Name', 'CriticActionFC1')
    reluLayer('Name', 'ActorRelu1')
    fullyConnectedLayer(NumNeuron, 'Name', 'CriticActionFC2')];


commonPath = [
    additionLayer(2,'Name', 'add')
    reluLayer('Name','CriticCommonRelu')
    fullyConnectedLayer(1, 'Name', 'output')];

criticNetwork = layerGraph(statePath);
criticNetwork = addLayers(criticNetwork, actionPath);
criticNetwork = addLayers(criticNetwork, commonPath);    
criticNetwork = connectLayers(criticNetwork,'CriticStateFC2','add/in1');
criticNetwork = connectLayers(criticNetwork,'CriticActionFC2','add/in2');

% set some options for the critic
criticOpts = rlRepresentationOptions('LearnRate',learing_rate,...
                                      'GradientThreshold',1);

% create the critic based on the network approximator
critic = rlQValueRepresentation(criticNetwork,obsInfo,actInfo,...
    'Observation',{'state'},'Action',{'action'},criticOpts);
agent = rlDQNAgent(critic,agentOpts)

trainingStats = train(agent,env,trainOpts);

After training, I'd like to get the network's trained weights and biases.

Answer 1

[Anh Tran]

You can get the parameters from the trained's critic representation for DQN agent. In MATLAB R2020a, see getLearnableParameters and getCritic functions (function name changes a bit since R2019b). You can follow similar steps to get the actor's parameters from actor-based agent like DDPG or PPO.

critic = getCritic(agent);
criticParams = getLearnableParameters(critic);

[rakbar]

Great, thank you! I didn't realize this was in the documentation already.

Thanks!

[rakbar]

Is there a way to get an Agent's weights and bias after each training episode?

[Dmitriy Ogureckiy]

the same question like above! anyway thank you for the answer !

[Dmitriy Ogureckiy]

Hi, Anh Tran, anyway, I don't understand how this function get this parameters, if agent don't have any weights ?

[Francisco Serra]

@rakbar @Dmitriy Ogureckiy have you found a way of getting the weights after each training episode?

[轩]

@Francisco Serra I have the same need. I find a silly method: save the agent after each episode and use "getLearnableParameters" to print the parameter of each agent.