Is it possible to achieve centralized learning in multi-agent RL with a customize function layer which is different among agents?

I am trying to do multi-agent reinforcement learning in MATLAB. To achieve centralized learning process, all the agents have the same observation and action. Moreover, their neural networks are almost the same in addition to a customized function layer. The function layer is used to extract the corresponding sub observation from the whole observation for each agent, and hence is different among agents. I understand that I can give different observations to different agents so as to avoid such a function layer. However, as different agents have different observations, i.e., the dimension of observation, while the centralized learning process requiring all the agents to be identical, such a modification prohibits the centralized learning process.

So, may I ask what I can do after putting such a function layer in the neural network if I still want to go with the centralized learning? Honestly, the aforementioned function layer does not contain any learnables.

Answer 1

[UDAYA PEDDIRAJU]

Hi Gavid,

Centralized learning with different observations is tricky. Here are options for your custom function layer setup:

1. Preprocess for Uniformity: Standardize observations before feeding them to the network. This allows centralized learning while retaining relevant information.
2. Agent Embeddings: Use the function layer to create agent-specific embeddings within a unified observation space.
3. Actor-Critic (Centralized Critic): Use an Actor-Critic architecture with a shared critic network for centralized learning. Each agent's actor network would process observations with the custom layer before feeding them to the critic.
4. Centralized Critic (Decentralized Actors): Alternatively, use a centralized critic network but have decentralized actor networks with the custom function layer.

In order to achieve centralized learning in multi-agent RL with a customized function layer that is different among agents, you can use MATLAB's Deep Learning Toolbox. Here's an example of how you can implement this:

% Define the neural network architecture
layers = [
    imageInputLayer([numObservations, 1, 1])
    fullyConnectedLayer(64)
    reluLayer
    fullyConnectedLayer(numActions)
    softmaxLayer
    classificationLayer
    customFunctionLayer % Add your customized function layer here
];

% Create the network for each agent
numAgents = 5;
networks = cell(numAgents, 1);
for i = 1:numAgents
    networks{i} = createNetwork(layers);
end

% Train the networks using centralized learning
% ...

% Update the networks using centralized learning
% ...

% Use the networks for inference
% ...

In this example, "numObservations" represents the dimension of the observation, and "numActions" represents the dimension of the action. The "customFunctionLayer" is where you can define your customized function layer that extracts the corresponding sub-observation for each agent.

By creating separate networks for each agent with the same architecture but different function layers, you can achieve centralized learning while accommodating the different observations of each agent.

You can refer the following documentation of deep learning toolbox for more details: https://www.mathworks.com/products/deep-learning.html