How do I view validation data when using the Bayesian Regularization Algorithm to train a neural network?

When using the Bayesian Regularization training algorithm in the Neural Fitting tool (nftool), why are the MSE and R2 values not present for the validation set? I would like to view the MSE and R2 values of the validation set of his data.

Reproduction steps:

I have followed the tutorial at the following link, using the Bayesian Regularization training algorithm:

https://www.mathworks.com/help/deeplearning/gs/fit-data-with-a-neural-network.html

Notice that the Neural Fitting tool's results for validation has a MSE and R2 value of 0. In addition, the Neural Network Training's (nntraintool) plot of performance and regression both lack plots of the validation data.

Answer 1

[MathWorks Support Team]

The function 'trainbr' that performs Bayesian regularization backpropogation disables validation stops by default. The reasoning for this is that validation is usually used as a form of regularization, but 'trainbr' has its own form of validation built into the algorithm. More information on the 'trainbr' function can be found at the documentation link below:

 

https://www.mathworks.com/help/deeplearning/ref/trainbr.html

If we have a network called "net", this behavior of validation stops is controlled via the parameter 'net.trainParam.max_fail', which gets set to 0 by default by 'trainbr' (see documentation link above). 'max_fail' denotes the maximum number of times that we allow the validation to improve to not improve before terminating training. 

If we set 'max_fail' to 5, the training will terminate when we get 5 consecutive iterations where the validation performance does not improve. If we want to get the validation results without terminating the training, we can set 'max_fail' to a very large number like 10000.

Unfortunately, 'max_fail' cannot be set via the neural network app. But, we can use the app to generate a script for the network, and then edit the script to set 'max_fail' to a large number. Then, running this script will allow you to get validation results. I have attached an example script generated from the app where "max_fail" is set to a large number (see line 37 of the attached script).

Note that the script assumes that two variables, 'simplefitInputs' and 'simplefitTargets' are set to the input data and target data, respectively. When I ran the attached script, I used the sample 'chemical_dataset' by executing the following in the MATLAB Command Window:

>> load chemical_dataset
>> simplefitInputs = chemicalInputs
>> simplefitTargets = chemicalTargets
>> % run bayesian_regularization_validation.m script

After running the attached script, we can now view the validation and regression plots, noticing that the validation data is present.

[Greg Heath]

It is possible to use

BOTH VALIDATION AND REGULARIZATION

 HOWEVER, IT IS NOT A MATLAB OPTION.

I recall modifying the training algorithm so that both can be used at the same time.

However, I neither remember details or where (NEWSGROUP or ANSWERS?) I posted it.

If you find it please let us know.

Greg

[Greg Heath]

I think this may be it:

https://www.mathworks.com/matlabcentral/answers/405727-why-trainbr-function-does-not-require-a-validation-dataset

Greg