Experiment Manager extremely slow

Hello,

when I train a neural network it takes about 5-7 mins of computing time on my pc. When I compute it via experiment manager (bayesian optimization) it takes an hour to compute the network with the same stats... why such a huge difference?

Thanks in advance

Best

[晓龙 张]

I have the same problem. The time is 2 times longer even worst! I wonder how to solve this problem?

[Wendimu]

I am also having the same problem even with Shallow Neural Networks! What could be the cause of this?

Answer 1

[Matt J]

But you understand, right, that when you use the Experiment Manager, your network is being retrained multiple times with different sets of hyperparameters? If as part of its search, the EM needs to test 10 sets of parameters, each requiring 5-10 min. of training, it makes sense that the whole exercise would be an hour long or more.

[ThomasP]

yes I know, I use bayesian optimization with a max amount of 30 trials, I don't mean that those 30 trials take an hour, I mean the training of one set of hyperparameters takes one hour, until all 30 are finished it will take 30 hours

I've added a picture where you can see that a single trial with specific hyperparameters takes one hour

[Matt J]

Can you rig the Experiment Manager to execute a single trial, consisting precisely of the hyperparameter set which normallly takes you 5-7 min? Does it still take longer when you run that as an EM trial?

[晓龙 张]

That is not the reason. The speed is just very low even when single experiment is conducting!

[Matt J]

But the training time will be different for every experiment, depending on the values of the hyperparameters. Have you really done a side-by-side comparison of the exact same training experiment, both with the same choice of hyperparameters?