Is there a universal learning rate for NeuralNetworks?

Question

I'm currently creating a NeuralNetwork with backpropagation/gradient descent. There is this hyperparameter introduced called "learning rate" (η). Which has to be chosen to guarantee not overshooting the minimum of the cost function when doing gradient descent. But you also do not want to slow down learning unnecessarily. It's a tradeoff. I've found that for too small or too big η the NeuralNetwork doesn't learn at all.

I've successfully trained the NN on the sin-function with η = 0.1. But for other functions like any linear combination of the inputs, a different η is required (more like η = 0.001). For the quadratic function, I still haven't been able to make the NN converge at all, maybe I just haven't found the right hyperparameters.

My question now is: Is there any way I can find a η that works for any function, so I don't have to try and search for it manually.

Thanks in advance, Luis Wirth.

Answer 1

There is no universal learning rate. It depends on your problem space (Are you solving a problem with many local minima or just one? Does your problem’s solution vary dramatically based on a slight change to your input or does the solution gradually shift?, etc.), and it depends on your network architecture (number of layers, number of hidden layers. Do you have a feedback loop etc).

Character recognition is relatively easy for example so you could try a faster learning rate. I once tried to teach a neural network to understand a simple computer machine code, and after a month of training with a dozen computers, it always came close but never solved the problem 100% correctly. No choice of learning rate worked to get me to the solution.

Once I broke down the problem space into a few subcomponents, the entire set of neural networks were trained in under an hour. I used the same choice of training rate in both cases (see below). A good example of how the rate of convergence changes based on the problem being trained.

What I found works best is to start with a fast learning rate to quickly find local minima, but then reduce the rate with each successive training, down to some limit.

All that to say that there is no universal learning rate, and that it worked best for me to reduce it as the training proceeds.