Neural network diverging instead of converging

Question

I have implemented a neural network (using CUDA) with 2 layers. (2 Neurons per layer). I'm trying to make it learn 2 simple quadratic polynomial functions using backpropagation.

But instead of converging, it is diverging (the output is becoming infinity)

Here are some more details about what I've tried:

• I had set the initial weights to 0, but since it was diverging I have randomized the initial weights (Range: -0.5 to 0.5)
• I read that a neural network might diverge if the learning rate is too high so I reduced the learning rate to 0.000001
• The two functions I am trying to get it to add are: 3 * i + 7 * j+9 and j*j + i*i + 24 (I am giving the layer i and j as input)
• I had implemented it as a single layer previously and that could approximate the polynomial functions better than it is doing now
• I am thinking of implementing momentum in this network but I'm not sure it would help it learn
• I am using a linear (as in no) activation function
• There is oscillation in the beginning but the output starts diverging the moment any of weights become greater than 1

I have checked and rechecked my code but there doesn't seem to be any kind of issue with it.

So here's my question: what is going wrong here?

Any pointer will be appreciated.

Answer 1

1. With neural networks, you always need to randomly initialize your weights to break symmetry.
2. If you don't use a non-linear activation function in the hidden units, then you might as well have stayed with a single layer. Your network is now just a composition of two linear functions, which is of course just another linear function.
3. That learning rate seems excessively small. If I'm using a fixed learning rate I normally find a value somewhere between 0.1 and 0.0001 to work well for most problems. This is obviously problem dependent so take my experience for what it is.
4. In addition to checking your code against the math you've learned, when doing gradient based optimization it can be very helpful to analytically calculate the gradients required using finite differences and compare them to the values you're computing in your code. See here for a discussion of how you can do this. I've caught many errors this way and seeing these types of tests pass always helps me feel much more sure of code correctness.