using fminsearch to optimize SVM parameters

Hi

I am trying to use fminsearch to optimize the following SVM parameters:

rbf_sigma boxconstraint

It seems that fminsearch is always getting stuck in a local minimum and giving me solutions far from the optimal solution.

Can you tell me if there is something wrong in my code causing this problem?

c= cvpartition(d1,'kfold',10);

minfn = @(z)crossval('mcr',Xs,Y,'Predfun', @(xtrain,ytrain,xtest)cross_fun(xtrain,ytrain,xtest,exp(z(1)),exp(z(2))),'partition',c);

opts = optimset('TolX',5e-4,'TolFun',5e-4);

[searchmin fval1] = fminsearch(minfn,randn(2,1),opts);

where the function "cross_fun" is defined as:

function [ yfit ] = cross_fun(xtrain,ytrain,xtest,rbf_sigma,boxconstraint)

svmStruct = svmtrain(xtrain,ytrain,'Kernel_Function','rbf','rbf_sigma',rbf_sigma,'boxconstraint',boxconstraint); yfit = svmclassify(svmStruct,xtest);

end

note: when I tried to track how fminsearch is changing the values of rbf_sigma boxconstraint I noticed it's not changed it far from the initial values. example: initial value is 1.5 its search around 1.505, 1.495 while the optimum value is actually around 7

Answer 1

[Shashank Prasanna]

I haven't run your code with sample data, but it is true that FMINSEARCH will always look for local minima, and the only way to break away from this is to run the same optimization problem with several different starting points and hope that you find what you are looking for.

In theory the only way you can be confident your minima is global if there there is an analytical solution or if you evaluate every single point on your objective surface.

That being said you MATLAB has a Global Optimization Toolbox, which offers GA, PATTERNSEARCH and other optimization techniques which are great approaches to find global minimum. You can check them out here:

http://www.mathworks.com/products/global-optimization/

http://www.mathworks.com/help/gads/index.html

[Hadi Masoud]

Thanks Shashank for your reply

I know that fminsearch does not find a global minimum however i'm not sure if the code is being stuck in a local minimum or just being stuck around whatever the initial value is.

Using my current code there no advantage in using fminsearch i might as well just try different values without using fminsearch

[Shashank Prasanna]

Hi Hadi, a thing that concerns me about your objective function is that you are cross validating inside it. Which means you objective surface is changing at each iteration due to the randomness of the objective function. At each iteration fminsearch is trying to minimize a completely different objective.

You may want to try calculating the misclassification rate for the same set of data (instead of cross validating) and find the best rbf_sigma and boxconstraints. See if that helps to start with.

[yem]

hi i have a system with 2 delays i want to identify these delays with optimisations tools(Newton algorithm)

Answer 2

[Alan Weiss]

As I mentioned on comp.sys-soft.matlab, this code is taken directly from a documentation example. And you would probably do well to begin your search for a global minimum from a wide variety of initial points, such as

y = logspace(-3,3,10);
[xx,yy] = meshgrid(y);
z = [xx(:),yy(:)];

Use z as the start of 100 different local searches--fminsearch will still get stuck in local minima, but you have a lot of starting points to search through.

And as Shashank mentioned (and also mentioned in the documentation example), using patternsearch as the local optimizer will probably work much better than fminsearch.

Good luck,

Alan Weiss

MATLAB mathematical toolbox documentation