¿La sequenceInputLayer de matlab agrega infomación posicional? Necesito agregar información posicional para crear mi transformer pero no encuentro un bloque que haga eso

estoy creando la arquitectura de red de mi transformer pero no se como poner la información posicional con Deep Network Designer, este es es código que tengo hasta ahora...

function lgraph = setNeuralNetworkArchitecture(inputSize, numClasses)

% Create layer graph

lgraph = layerGraph();

% Add layer branches

tempLayers = [

sequenceInputLayer([13 24 8],"Name","sequence")

% Añadir positional encoding

lambdaLayer(@(x) addPositionalEncoding(x, encodingDim), 'Name', 'positionalEncoding')

flattenLayer("Name","flatten")];

lgraph = addLayers(lgraph,tempLayers);

tempLayers = selfAttentionLayer(16,64,"Name","selfattention");

lgraph = addLayers(lgraph,tempLayers);

tempLayers = [

additionLayer(2,"Name","addition")

fullyConnectedLayer(6,"Name","fc_1")

softmaxLayer("Name","softmax")

classificationLayer("Name","classoutput")];

lgraph = addLayers(lgraph,tempLayers);

% clean up helper variable

clear tempLayers;

lgraph = connectLayers(lgraph,"positionalEncoding","flatten");

lgraph = connectLayers(lgraph,"flatten","selfattention");

lgraph = connectLayers(lgraph,"flatten","addition/in2");

lgraph = connectLayers(lgraph,"selfattention","addition/in1");

end

Answer 1

[Aiswarya]

Tenga en cuenta que responderé a la pregunta en inglés

(Please note that I will be answering the question in English)

Hi Luis,

It is my understanding that you are creating a transformer network and you wish to add a positional encoding layer to your network. You want to know if it can be done by the sequenceInputLayer or if there is some other way to do it.

There is no particular layer to add positional encoding in Deep Network Designer. The following list contains all deep learning layers in MATLAB :

List of Deep Learning Layers - MATLAB & Simulink (mathworks.com)

However, you can create a custom layer for positional encoding. This can be added as an intermediate layer with the help of templates provided in the following documentations:

Define Custom Deep Learning Layers - MATLAB & Simulink (mathworks.com)

Define Custom Deep Learning Layer with Formatted Inputs - MATLAB & Simulink (mathworks.com)

This code snippet can give you an idea how to create the positional encoding layer:

classdef PositionalEncodingLayer < nnet.layer.Layer
    properties
        SequenceLength
        NumFeatures
    end
    
    methods
        function layer = PositionalEncodingLayer(sequenceLength, numFeatures)
            layer.SequenceLength = sequenceLength;
            layer.NumFeatures = numFeatures;
            layer.Name = 'PositionalEncodingLayer';
        end
        
        function Z = predict(layer, X)
            positionEncodings = layer.getPositionalEncodings();
            Z = X + positionEncodings;
        end
        
        function positionEncodings = getPositionalEncodings(layer)
            % This describes the position encoding algorithm 
            position = (1:layer.SequenceLength)';
            divTerm = 10000.^((0:2:layer.NumFeatures-1)./layer.NumFeatures);
            positionEncodings = zeros(layer.SequenceLength, layer.NumFeatures);
            positionEncodings(:, 1:2:end) = sin(position * divTerm);
            positionEncodings(:, 2:2:end) = cos(position * divTerm);
        end
    end
end

I hope this helps.

Regards,

Aiswarya