I'm using divergence here as to mean that the gradient is getting further and further from zero in stochastic gradient descent. I've written my own feed-forward neural network and tried to use it to solve several incredibly basic problems of binary classification such as:
- Being given two random real numbers from zero to one and having the first neuron in the output layer have a higher net output if the two numbers add up to be greater than 1 and else having the second neuron have a higher net output.
- The same problem as the first but the two numbers have to add up to at least 2 (which is impossible).
In the case of both problems the network does succeed at solving them within a couple of seconds. Since it's binary classification I made a way of visualising the data similar to Tensor Flow Playground which you can see in the following image:
The light blue area represents area which the network believes has met the condition
The dark area is area that according to the network has not met the condition
The red and blue circles represent training data. (There are 200 individual pieces of training data). The red circles are ones that fail to meet the criteria and vice-versa.
The error at the bottom is updated after every backpropagation step.
Weights and biasses are randomly generated between 1 and -1 and activation functions are linear. Batch size is 50 and backpropogation happens 10 times per second. Quadratic Error function is used.
Once the network solves the problem (or gets quite close, better than in the picture.) The error stagnates for a second or two before jumping to something massive like 400 (When I say jumping, it goes up really fast but it does climb it's not instantaneous). Then over a second it works its way back down to 1. After a second it works its way back down to something manageable like the error in the picture. Then this continues several times until eventually the error climbs a bit too high goes past infinity and ends up as NAN.
With reference to the visualisation: The line between the dark and the light-blue jumps back and forward rapidly from either side only once it gets close to the answer. Then it either settles back down close to the solution or the solution or if the error goes to NAN it goes entirely light blue or entirely dark.
Obviously the weights and biasses follow suit. It appears to me that the network is overshooting steps once the error gets really small, making the error bigger, the gradient bigger, the weights bigger, the biasses bigger and thus the next error bigger. If by chance this continues it will eventually snow ball in to an unstoppable divergence.
Before you say, "make your learning rate smaller!":
I've tried learning rates of 0.0001 to no prevail, it just takes a bit longer to go wrong. I've also tried a learning rate that scales down as the error does:
return (error > 0.7) ? 0.05 : Math.pow(error,0.8) / 14;
My question here is what could be wrong? Based on my paragraph where I tried to explain what I thought the problem was myself is there something wrong with my fundamental understanding of how backpropgation should work that I could have transferred into my algorithm? My biggest question is why doesn't this always happen in people's neural networks. How do you stop divergence inevitably happening be sheer probability? What part of this am I not correctly understanding?
If you read all this I'm eternally thankful... I'd love if you'd share whatever knowledge you have. Thanks in advance.