how to optimize two variables with different scales

I am using fminsearch function to optimize two variables, say, a and b. They are the values returned from the optimization function.

a and b however, are in different scales, a is of 10^-1, while b is of 10^1. if I want to find minimum of a and maximum of b, is it okay I just return a-b from the optimization function? will different scales make the optimization worse?

And another concern is the tolerance I set in the options argument from 'fminsearch',TolX and TolFun. How could I set the tolerance as relative one instead of absolute? The reason is that all my tuning parameters and optimize targets are of different scales.

Answer 1

[Walter Roberson]

a-b would work in theory if the influence of a and b are independent. As soon as there is cross-effect than a-b would imply that a change of 1% in b would of the same importance as a change of 100% in a. That probably is not what you want.

[jin wang]

Hi, thank you for the reply. Yes, they are coupled in a way. Actually a is the standard deviation, and b is the percentage of the prediction falls inside +- std area.

How can I optimize them both at the same time? Can I try to normalize them first before optimization?

[Walter Roberson]

The larger your sample size the closer the percentage within a given standard deviation will come to being statistically constant, so I would wonder if it is a good thing to optimize or if instead you would end up optimizing statistical accidents?

It is not immediately clear to me that the standard deviation can be usefully normalized more than it already is.

If you have small sample sizes then the percentage you mention could end up badly quantized.

[jin wang]

Thank you for the reply. My data set is not big, and finding the standard deviation so far does has some effect on my model performance. My recent doubt is when I am using the fminsearch function:

Let's say the tolerance I am using is for searching the minimum of a returned value from optimization function.

I have stated the tolerance inside fminsearch function as options argument. How can I know the value I set is relative or absolute tolerance? How can I set a relative tolerance? since I don't know if the auto tolerance setting updates in 2017a onwards has affected fminsearch or not.