Does the Neural Network Toolbox have a Cross Entropy Error Function?

I would like to know if the Neural Network Toolbox has a Cross Entropy Error Function.

Answer 1

[MathWorks Support Team]

Starting in R2013b, Neural Network Toolbox provides a CROSSENTROPY function:

https://www.mathworks.com/help/deeplearning/ref/crossentropy.html

Prior releases of Neural Network Toolbox do not provide a Cross Entropy Error performance functionality. This is an alternative to the mean-squared error (MSE) function. In general, the values of 'd', the desired output for a given training example, and 'y', the observed output for that case, will be real numbers. We can interpret 'd' as the desired probability that a binary-valued output will assume a value of 1 in this case, and 'y' as the observed probability of seeing a 1 in that case.

The cross-entropy error C is then expressed as:

C = - sum [all cases and outputs] (d*log(y) + (1-d)*log(1-y) ) 

The derivative of this error function for a given output and training example (this is the value we actually back-propagate) is:

dC/dy = - d/y + (1-d)/(1-y)

Note that this back-propagated derivative goes to infinity as the difference between y and d goes to +1 or -1. This can counteract the tendency of the network to get stuck in regions where the derivative of the sigmoid function approaches zero.

Refer to the Neural Network Toolbox User's Guide, specifically to the chapter on custom networks, for information on how to set a custom performance function through the 'performFcn' property:

https://www.mathworks.com/help/deeplearning/ref/network.html

Additional information may also be found in the related solutions below.

[Greg Heath]

Also see the following NEWSGROUP threads:

2 May 2009 Estimation of Multivariate Classification Posterior Probabilities Greg Heath 13 3951

6 Apr 2009 Unbalanced Priors in BioID Face Image Data Greg Heath 54 5009

14 Apr 2006 NN performance function Greg Heath 3 756