How is the Delta Rule derived in neural networks and what is the explanation for the algebra?

Question

I am currently trying to learn how the Delta Rule works in a neural network. So far I completely understand the concept of the Delta Rule, but the derivation doesn't make sense. My question is how is the Delta Rule derived and what is the explanation for the algebra. Currently I am writing equations to try to understand, they are as follows:

Desired = (Input * Weight_D)

Actual = (Input * Weight_A)

Error = (Input * Weight_D) - (Input * Weight_A)

Error = Input (Weight_D - Weight_A)

However, as stated Here, the ΔWeight is understood to be Error * Input, where as I derive it to be Error / Input. I fully understand the concept of correcting the weights in a neural network, however, I do not understand how the algebra and derivation of the Dela Rule work to re-adjust the weights to correctly represent the output. I will explain my current understanding to help you understand why I may be confused. Because the Input is constant whilst you attempt to find a correct weight, you can factor it out. This leads me to the equation that the Error is equal to the Input times the difference between the correct and incorrect weights. This is also very clear. However I do not understand where they go from here to get the equation to correct the weights. How is the required net change derived from this? How does the algebra behind simply Input * Error work? Thank you.

Answer 1

Your equations are wrong on several counts. First of all, the weights and the inputs are both vectors. Second, the value of actual is given by $\operatorname{sgn}(\langle \mathit{Input}, \mathit{Weights} \rangle)$, in words, the sign of the inner product of the input and the current weights. Similarly, the desired output is either $+1$ or $-1$. Therefore the error is either $+2$ (if the desired was $+1$ but the actual was $-1$) or $-2$ (in the opposite case). Third, we don't want to "solve" for the weights, since this would erase everything that we have learned so far.

The idea of the learning process is to give negative feedback. Suppose that we got $-1$ but we expected $+1$. We want to change the weights so that on the same input, $\langle \mathit{Input}, \mathit{Weights} \rangle$ will be larger. The most economical way (in some sense) is to update the weights with a new vector proportional to the input, since the input is the vector most correlated with itself (that is, the unit vector $w$ maximizing $\langle \mathit{Input},w \rangle$ is proportional to $\mathit{Input}$). Thus we increase the weights with some multiple of the input. If the error were in the other direction, we would decrease instead of increase — which is why we multiply the input by the error.