Custom loss function for DNN training

I have the feed forward network as shown bellow. It consists of an input sequence layer, 2 hidden fully conected and an output regression layer.

Network architecture:

I have defined the layers as follows:

layers = [

sequenceInputLayer(4,"Name","sequence")

fullyConnectedLayer(4,"Name","fc_1")

fullyConnectedLayer(4,"Name","fc_2")

regressionLayer("Name","regressionoutput")];

I need to train the model where the input will be the same as the output. I need to restrict the weights of the layers using the following loss function:

L=L1+L2+L3+L4

where

L1= λ|cov(H)-I|

L2= λ|(H/4)+λ|W1*transpose(W1)-I|

L3= λ|cov(Q)-I|

L4= mse of input-output

H is the output of the 1st hidden layer and Q is the output of the second hidden layer and W the weight of each layer.The total loss function is L=L1+L2+L3+L4.

What is the best approach for this? Could posibly be done with regularisation?

Thank you!

Answer 1

[Pratyush Swain]

Hi Fotios,

I understand you want to create a custom deep learning network.Please refer to https://in.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layers.html. It contains information regarding custom 'Regression output layer' and 'Intermediate layer'.

The output layer uses two functions to compute the loss and the derivatives: forwardLoss and backwardLoss. The forwardLoss function computes the loss L. The backwardLoss function computes the derivatives of the loss with respect to the predictions.

For example: To write a weighted cross entropy classification loss, try running this in the MATLAB command window

>>  edit(fullfile(matlabroot,'examples','deeplearning_shared','main','weightedClassificationLayer.m'))

Hope this helps.

Answer 2

[David Ho]

Hello Fotios,

You can solve this constrained learning problem using a dlnetwork object and a custom training loop:

https://uk.mathworks.com/help/deeplearning/ug/train-network-using-custom-training-loop.html

If I understand your question correctly, I believe the loss function should look something like this:

layers = [
    sequenceInputLayer(4,"Name","sequence")
    fullyConnectedLayer(4,"Name","fc_1")
    fullyConnectedLayer(4,"Name","fc_2")];

net = dlnetwork(layers);

% Test loss function with random inputs and targets
rng(0);
X = dlarray(rand(4,10), "CT");
T = dlarray(rand(4,10), "CT");
lambda = 1;
loss = dlfeval(@(net,X,T) modelLoss(net, X, T, lambda), net, X, T)

loss = 
  1×1 single dlarray

   58.8300

grad = 4×3 table
    Layer     Parameter        Value    
    ______    _________    _____________

    "fc_1"    "Weights"    {4×4 dlarray}
    "fc_1"    "Bias"       {4×1 dlarray}
    "fc_2"    "Weights"    {4×4 dlarray}
    "fc_2"    "Bias"       {4×1 dlarray}

function [loss, grad] = modelLoss(net,X,T,lambda)
% Get the activations and weights of the fully connected layers
[h,q] = forward(net, X, Outputs=["fc_1", "fc_2"]);
w1 = net.Layers(2).Weights;

% Compute covariance matrices
covH = cov(stripdims(h)');
covQ = cov(stripdims(q)');

% Compute components of loss function
l1 = dlNorm(lambda*covH - eye(size(covH)));
l2 = lambda*dlNorm(h/4 + lambda*norm(w1*w1' - eye(size(w1)), "fro"));
l3 = lambda*dlNorm(covQ - eye(size(covQ)));
l4 = mse(q,T);

% Compute loss and gradients
loss = l1 + l2 + l3 + l4;
grad = dlgradient(loss, net.Learnables);
end

function normX = dlNorm(X)
% Frobenius norm of a dlarray
normX = sum(stripdims(X)'*stripdims(X), "all");
normX = sqrt(normX);
end