I'm currently creating a NeuralNetwork with backpropagation/gradient descent. There is this hyperparameter introduced called "learning rate" (η). Which has to be chosen to guarantee not overshooting the minimum of the cost function when doing gradient descent. But you also do not want to slow down learning unnecessarily. It's a tradeoff. I've found that for too small or too big η the NeuralNetwork doesn't learn at all.
I've successfully trained the NN on the sin-function with η = 0.1. But for other functions like any linear combination of the inputs, a different η is required (more like η = 0.001). For the quadratic function, I still haven't been able to make the NN converge at all, maybe I just haven't found the right hyperparameters.
My question now is: Is there any way I can find a η that works for any function, so I don't have to try and search for it manually.
Thanks in advance, Luis Wirth.