How can I learn the final learning rate of a deep neural network?

After completing the training procedure of a deep neural network over a spesific dataset, how can we learn the last, final, optimum learning rate? We specify a certain learning rate at the beginning, it is OK, but it reaches to an optimum value? What is it? How can I learn it?

Could you please refer a sample test case below:

clc;clear;
[XTrain,YTrain] = japaneseVowelsTrainData;

figure
plot(XTrain{1}')
title("Training Observation 1")
numFeatures = size(XTrain{1},1);
legend("Feature " + string(1:numFeatures),'Location','northeastoutside')

inputSize = 12;
numHiddenUnits = 100;
numClasses = 9;

layers = [  sequenceInputLayer(inputSize)
            lstmLayer(numHiddenUnits,'OutputMode','last')
            fullyConnectedLayer(numClasses)
            softmaxLayer
            classificationLayer]

maxEpochs = 70;
miniBatchSize = 27;

options = trainingOptions('sgdm', ...
            'ExecutionEnvironment','cpu', ...
            'MaxEpochs',maxEpochs, ...
            'MiniBatchSize',miniBatchSize, ...
            'GradientThreshold',1, ...
            'Verbose',false, ...
            'Plots','training-progress');

net = trainNetwork(XTrain,YTrain,layers,options);
[XTest,YTest] = japaneseVowelsTestData;
YPred = classify(net,XTest,'MiniBatchSize',miniBatchSize);
acc = sum(YPred == YTest)./numel(YTest)

Answer 1

[Meet]

Hi Faruk,

From my understanding of your question, you want to know the final learning rate at the end of training the deep neural network.

There are two ways to do so, let me provide both the ways,

1. Training Progress Window: You can find the final learning rate which was used by the network. Since, the learning rate schedule is constant it would remain 0.01, even at the end of the training.
2. Using ‘info’ object: ‘trainNetwork’ method return two objects one is ‘net’ and other is ‘info’ which is a struct and contains an array of learning rates named ‘BaseLearnRate’, you can access the last element of this array to know the final learning rate the model used­­­­. In your case you could replace the line

     net = trainNetwork(XTrain,YTrain,layers,options);

to

     [net,info] = trainNetwork(XTrain,YTrain,layers,options);
     finalLearningRate = info.BaseLearnRate(end);

I hope these suggestions prove helpful for your task!

You can refer to this article for more detailed information:

https://www.mathworks.com/help/deeplearning/ref/trainnetwork.html