Are Regression Models Deterministic?

I have a script where I rebuild a Bi-Level Neural Network Regression Model on a new Training data set, and then use the new model to predict on a prediction data set. The original Regression Model (in this case a Bi-level Neural Network) is built in the Regression Learner, and then the Function is generated and saved.

The issue is that when I rebuild the model using the same training data set, and then predict using the same prediction data set, the model produces different results.

I'd like to understand why this is the case. I would have thought that Regression Models would be deterministic, in other words given the same training data to learn on, and then passing the same prediction data set, I would get the same results. But, this is not happening.

Here is my sequence:

• I construct a Bi-Level Neural Network in the Regression Learner
• I then generate, export and save the Model Function as BNN_Model.m

This is done only once. I then use BNN_Model.m in a script that will rebuild new prediction models using different Training data sets:

• The function is used to create a new model using an imported training table >> NewBNN_Model = BNN_Model(TrainingTable)
• This creates NewBNN_Model.mat
• This new model is then used to predict using an imported prediction table >> ResultTable = NewBNN_Model.predictFcn(PredictTable)

The natue of my application is such that occasionally the same TrainingTable and PredictTable are sent to the script. When this happens my expectation is that the ResultTable should be identical. But, it is not.

I created a simple test sequence where the same TrainingTable and PredictTable are sent to the script for 10 consecutive iterations. All 10 ResultTables have different values. In some cases radically different values. In this simple test, the TrainingTable has 40 rows and the PredictTable has only 1 row.

Can anyone explain why this is happening? Are Regression Models non-deterministec, and I should expect different results? Or, am I doing something wrong?

Answer 1

[dpb]

By default, the toolset uses bayesopt so the selection is, indeed, randomized. Unless the data are virtually noiseless, one can expect different (and as you've noticed, occasionally widely so) results.

You would have to save the random number generation state and feed it back to the regression learner app to reproduce the same results, or use a non-Bayesian alternative that is not randomized.

[dpb]

"...save the random number generation state and feed it back to the regression learner app to reproduce the same results,"

More specifically, save the state of the random number generator before starting the regression with

sBeginning=rng;     % return current rng state

and then execute

rng(sBeginning)

before the next time evaluate the fit. This, of course, means having to save the state array for every trial that want to repeat and reset to that specific state. The other alternative might be to use the same state for every calculation, thereby making it deterministic but dependent upon the specific state chosen -- and, as you've discovered that might or might not be a desireable result.