Questions about the regularization (Modified Performance Function) of neural network

Hello, everyone. I tried to find out the best regularization ratio for a very simple problem from Matlab, using the function trainbgf for a shallow neural network. Then I plotted a validation curve. The problem is that the curve didn't make any sense. I just followed the contents from the official document as follows:

Here are my codes.

*******************************************

regularization_term = [0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1];

m = size(regularization_term,2);

[x,t] = simplefit_dataset;

x_train = x(1:70);

t_train = t(1:70);

x_test = x(71:94);

t_test = t(71:94);

trainPerformance = zeros(50,11);

testPerformance = zeros(50,11);

for j = 1:50

for i = 1:m

net = feedforwardnet(10,'trainbfg');

net.divideFcn = '';

net.trainParam.epochs = 300;

net.trainParam.goal = 1e-5;

net.performParam.regularization = regularization_term(i);

net = train(net,x_train,t_train);

y_train = net(x_train);

trainPerformance(j,i) = sqrt(perform(net,t_train,y_train));

y_test = net(x_test);

testPerformance(j,i) = sqrt(perform(net,t_test,y_test));

end

end

plot(regularization_term, mean(trainPerformance),regularization_term,mean(testPerformance))

legend('trainperformance-RMSE','testperformacne-RMSE','best')

xlabel('Regularization Ratio')

ylabel('RMSE')

************************************************

Here is the learning curve I plotted.

I think that the RMSE of the training data should increase as the regularization ratio increases and the RMSE of the test data should decrease at first and at a certain point start to increase as the regularization ratio increases. I'm not sure where I made a mistake, can anyone give me advice? Thank you in advance!

[Greg Heath]

What is the URL of your

"OFFICIAL DOCUMENT" ???

Greg

[Wen Yui]

Thanks for your reply. Here is the link.

https://ww2.mathworks.cn/help/nnet/ug/improve-neural-network-generalization-and-avoid-overfitting.html

It is very wired that as the regularization ratio gets close to 1, both the training error and the test error decrease to almost 0. It don't make any sense for me. Thank you for your advice in advance!

Answer 1

[Greg Heath]

Oh! … O.K.

The simplefit_dataset is smooth with 4 interior local extrema. Therefore, you probably only need H = 4 hidden nodes.

More than H = 4 hidden nodes can be considered overfitting. So, if you use the default H = 10, you will have an overfit net and should implement a mitigation to

 PREVENT OVERTRAINING AN OVERFIT NET.

The most common mitigations are

 1. DO NOT OVERFIT 

 A. No. of unknown weights <= No. of training equations:

       Nw <= Ntrneq

 AND/OR

 B. Minimize weighted sum of SQUARED ERRORS AND SQUARED WEIGHTS 

    MSE + gamma * MSW

 2. DO NOT OVERTRAIN:

 Use a validation subset to implement EARLY STOPPING

Hope this helps.

Greg

[Greg Heath]

For an I-H-O net and N data points MATLAB defaults:

 Ntrn  = N-2*round(0.15*N) % ~ 0.7*N

 Ntrneq = Ntrn*O

 Nw      = (I+1)*H+(H+1)*O

 Ntrneq >= Nw  % No overfitting

 ==>

 H <= (Ntrneq-O)/(I+O+1)

Hope this helps.

Greg

[Wen Yui]

I tried another data set but got the same result that the validation curve didn't make any sense, and I tried to replace the codes like { trainPerformance(j,i) = sqrt(perform(net,t_train,y_train));}, a included function which was used to help us calculate the RMSE, with the codes like {testperformance(1) = sqrt(sum(test_e.^2)./size(test_e,2));} and the validation curves finally became the ones that they should.

[Wen Yui]

Anyway, thanks for your comments!

[Greg Heath]

DO YOU UNDERSTAND THE FOLLOWING MATLAB CONCEPT OF VALIDATION?

 N    = Ndes + Ntst    % DATA   = [ DESIGN TEST ]

 Ndes  = Ntrn + Nval   % DESIGN = [ TRAIN  VALIDATION ]

==>

 N  = Ntrn + Nval  + Ntst  % DATA = [ TRAIN  VALIDATION TEST ]

Therefore, your use of the term "validation" can be confusing to introductory MATLAB users.

Hope this helps

Greg

[Wen Yui]

I think I understand what you show me. Sorry for my vague explanation. I used the term 'validation curves' many times in this question without any context so you may feel confused. You can refer to this website which gives a very complete account of validation curves. The reason why I raised this question is that I wanted to plot a right validation curve. Thanks very much!

http://scikit-learn.org/stable/modules/learning_curve.html

[Greg Heath]

I understood perfectly what you were doing.

My concern was/is for the less experienced readers.

Greg

[Wen Yui]

Yes, you are right. I should make it clear that what I wanted to do. Thanks.

Answer 2

[Greg Heath]

% When I search the command line window for info on "regularization" :

>> help regularization

regularization not found.

Use the Help browser search field to search the documentation, or type "help help" for help command options, such as help for methods.

>> doc regularization

   % No answer!

>> lookfor regularization

 plotbr    - Plot network performance for Bayesian regularization training.
 trainbr   - Bayesian Regularization backpropagation.
 msereg    - Mean squared error with regularization performance function.
 msnereg   - Mean squared normalized error with regularization performance function.
 lasso     - Perform lasso regularization for linear regression on tall arrays.
lassoglm  - Perform lasso or elastic net regularization for a generalized linear model.
lasso     - Perform lasso or elastic net regularization for linear regression.

 % Therefore, it seems that for general 
 %(i.e., nonlinear) applications
 %
 %     USE TRAINBR WITH MSEREG
 %
 % I will let you do the rest. First see the results of the following commands

>> help trainbr

>> doc trainbr

Hope this helps.

THANK YOU FOR FORMALLY ACCEPTING MY ANSWER

Greg

[Wen Yui]

Thanks for your reply sincerely! I know trainbr and used it much. However, in my present study, I want to plot a validation curve to see how the regularization term affects the training error and the test error. Actually, in my codes (net = feedforwardnet(10,'trainbfg')) you can see I used the function ''trainbfg'' whose cost function could be msereg. So I can use msereg to change the regularization term. The problem is if I used the regularization term 1 I can get the training error and the test error to 0. It is incorrect with no doubt. So I an wondering I may make a mistake in my codes or I misunderstand the meaning of the regularization term. You can find the contents attached here from the link I gave you in the former comment. I know you have answered many questions on this forum and admire your efforts. I hope you can continue to give some advice if you have spare time. Anyway, Thanks very much!

[Wen Yui]

I regard the function trainbr as a function which can automatically determine the regularization term for us. It would be very useful in many cases. But I want to figure out how the regularization term affects the results. So I think trainbr does not help in my present study.