Using bayesopt instead of fmincon in Matlab example of "Solving Burgers PDE with Deep Learning

Hello everyone,

I am trying to understand how to solve PDE with PINNs using different optimization methods. in this example (https://www.mathworks.com/help/releases/R2022b/deeplearning/ug/solve-partial-differential-equations-with-lbfgs-method-and-deep-learning.html)

the fmincon and L-BFGS are used to optimize the hyperparameters of the PINN. Actually I am trying to solve this example using the bayesopt but I do not have clue how to replace fmincon.

I hava read the documentation of bayesopt

(https://www.mathworks.com/help/stats/bayesopt.html) and (https://www.mathworks.com/help/stats/optimizablevariable.html)

But what is not clear to me is how to make the hyperparameters of PINNs in this example like neurons, weights and biases to be optimizable variables?

Thanks in advance.

[Alan Weiss]

It is a poor idea to try to optimize a high-dimensional problem like this using bayesopt. The bayesopt function is not intended for high-dimensional problems. I would not use it to optimize a 100-dimensional problem, much less the over 3000 variables that I see in this example. How do you propose to create that many optimizeable variables? Really, what you propose is not sensible.

Furthermore, bayesopt is not a gradient-based algorithm, so cannot make use of derivative information. This means that even if you could somehow get the solver to run, it would be so horribly inefficient that you would never get any useful information out of it.

Whatever you are trying to do, I suggest that you look for a different way to do it.

Alan Weiss

MATLAB mathematical toolbox documentation

[Muhammad Mahmoud]

@Alan Weiss

Thank you for clarifying. I wrapped this example in a function that used bayesopt to optimize the number of layers and number of neurons, and I obtained a smaller value for each (four layers and seven neurons). Is bayesopt still inapplicable to this example in this case?

[Alan Weiss]

I don't know how many optimizable variables your model has now. If just a dozen or so then yes, you can use bayesopt, though I am not sure why you would want to. What do you hope to gain? bayesopt is much, much slower than fmincon and cannot use derivative information. Do you suspect that there are multiple local minima? If so, you would almost certainly be better advised to use MultiStart with fmincon instead of bayesopt. But do what you want.

Alan Weiss

MATLAB mathematical toolbox documentation

[Muhammad Mahmoud]

@Alan Weiss

In fact, I do suspect multiple local minimums, but not in the present case. I simply wish to use this example to solve another PDE of the "Advection-Diffusion-Reaction" type. Also, I do not know based on what criteria the number of layers and number of neurons in this example would be selected, so I used bayesopt to find the optimal combination of number of layers and number of neurons. Many thanks for your helpful advices and insightful suggestions.