How to implement a custom performance function in R2013b

Good input from Amaria in 2013 (<http://www.mathworks.com/matlabcentral/answers/64319>)

I've replicated the fundamental steps before making ANY changes to the copy of mse (mymse.m and associated copy of +mse package of functions). I found that the steps with the unmodified mymse.m function do not execute correctly.

Here's the steps I followed (per Amaria's post):

copyfile([matlabroot '\toolbox\nnet\nnet\nnperformance\mse.m'], 'mymse.m');
copyfile([matlabroot '\toolbox\nnet\nnet\nnperformance\+mse'], '+mymse');
[x,t] = simplefit_dataset;  
net = fitnet(10,'trainscg');
net.performFcn = 'mymse'; 
net = train(net,x,t);

The training executes but the results show there is no fitting. The resulting model is a flatline. Its a simple test to try. Anybody have success?

As a related test, I used the mae as follows:

net = fitnet(10,'trainscg');
net.performFcn = 'mae'; 
net = train(net,x,t);

And this trained successfully. So it appears the performance functions provided by the NN Toolbox work, but user-defined performance functions dont work using the procedure described above by Amaria.

I could be doing something wrong, but I've carefully tried several times and cannot get mymse to work. Again, mymse is unchanged code from mse.

Any help is very much appreciated!

Answer 1

[Craig]

The first two copyfile commands are great, they pull the necessary functions into your current working directories. From there if we execute and follow the debug information we can discover the following.

The 'nn7' argument can be fed to train() or to net() but it appears it is ONLY essential for evaluation using net(). This is because when we set net.performFcn = 'mymse' the nntoolbox will automatically pick up our performance function and will use it for training (you can see it listed in the nntraintool). HOWEVER, if you then try to call net() without using the 'nn7' argument it will output all zeros. For whatever reason training detects the correct net.performFcn and uses it, but the evaluation function does not. You can see this from the command line or from the nntraintool which both show incorrect outputs of zero if net is just called blindly without the 'nn7' argument (as it is if the push buttons in the nntraintool GUI are used without alteration).

Another interesting aspect is that examining the templates for the cost functions it appears as if nn7 (or 'nnet7') is only currently implemented with mse, mae, msesparse, sae and sse cost functions, assuming that is what the 'nnet7' is doing. I don't know this for sure at this time.

If we stop the code when running inside the 'mymse' function, we see that initially the INFO data is setup and made persistent, and this carries details about which functions to use in evaluating various calls to performance. Presumably, if we want to modify the cost function (or override it) then this involves adjusting all these functions suitably. So we need to examine how we need to modify the following subfunctions,

@mymse.apply @mymse.backprop @mymse.forwardprop @mymse.dperf_dwb

Each of these functions can be found beneath the '+mymse' directory we just copied to our working directory. There appear to be other related functions there too (such as perfwb.m and perfw_to_ew.m and it may be necessary to modify them as well for at least consistency sake).

Answer 2

[Chibuzo Nnonyelu]

Hi Calvin, I am interested in using a custom performance function in MATLAB NNtool and I have been searching to no success. Might you have gotten a solution to your problem? Please help me out, if you have. Thank you.

[Calvin]

Hello Chibuzo.

I have not received input on my question. So I have no solution to the problem I experienced trying to implement a custom performance function.

I learned after my post that the training function that works well for my applications (Levenberg-Marquardt backpropagation) is inherently based on the minimization of the mse (or sse). Therefore, networks trained with net.trainFcn = 'trainlm' must use either the mse or sse performance function.

However my example, which specifies the conjugate-gradient training function 'trainscg', does not work. That is, I cannot get Matlab to use my customized performance function even tho it is simply an unmodified copy of the default mse.m code. So it is still an unsolved problem.

I suggest you post your question to the forum. Maybe an expert will see it and respond.

Good luck. Cal

[Gokhan Demirkiran]

You shold use: net = train(net,x,t,nn7); nn7 is the key here! for more information: https://www.mathworks.com/matlabcentral/answers/64319-neural-network-how-to-use-a-custom-performance-function?s_tid=answers_rc1-1_p1_BOTH