Why do we need to take the derivative of the activation function in backwards propagation?

Question

I was reading this article here: https://towardsdatascience.com/how-does-back-propagation-in-artificial-neural-networks-work-c7cad873ea7.

When he gets to the part where he calculates the loss at every node, he says to use the following formula:

"delta_0 = w . delta_1 . f'(z) where values delta_0, w and f’(z) are those of the same unit’s, while delta_1 is the loss of the unit on the other side of the weighted link."

And $f$ is the activation function.

He then says:

"You can think of it this way, in order to get the loss of a node (e.g. Z0), we multiply the value of its corresponding f’(z) by the loss of the node it is connected to in the next layer (delta_1), by the weight of the link connecting both nodes."

However, he doesn't actually explain why we need the derivative term. Where does that term come from and why do we need it?

My idea so far is this:

The fact that the identity activation function causes the term to disappear is a hint. The node doesn't feed into the next exactly as is, it depends on the activation function. When the activation function is the identity, the loss at that node just passes to the next one based on the weight. Basically, you just need to factor in the activation function somehow, specifically in a way that doesn't matter when it's the identity, and of course the derivative is a way to do this.

The issue is that this isn't very rigorous, so I'm looking for a slightly more detailed explanation.

Answer 1

As a disclaimer, I didn't read that page, but I can certainly explain where the derivatives come from. The backpropagation algorithm is actually a variant of the gradient descent algorithm.

Think about a single function for the moment. Suppose you have a function which responds to a single input and a single weight: $f(w, x)$. We want to adjust $w$ so that it gives expected answers for various known values of $x$.

Then by Taylor's theorem:

$$f(w + \delta w, x) \approx f(w, x) + \delta w \frac{\partial f}{\partial w}(w, x)$$

Or to put it another way:

$$f(w + \delta w, x) - f(w, x) \approx \delta w \frac{\partial f}{\partial w}(w, x)$$

So if you find a value for $x$ which gives the wrong output, this gives you an estimate of how "wrong" $w$ is, and it's related to the derivative of $f$.

This idea extends to multi-variate calculus. If you have a vector of weights and inputs $\mathbf{w}$ and $\mathbf{x}$, then Taylor's theorem says:

$$f(\mathbf{w} + \delta \mathbf{w},\mathbf{x}) \approx f(\mathbf{w}, \mathbf{x}) + \nabla_{\mathbf{w}}f \cdot \delta \mathbf{w}$$

Where $\nabla_{\mathbf{w}}f$ is the gradient of $f$ with respect to the weights only.

Now let's think about your case, where the function $f$ has a specific form: it's a function applied to the dot product of $\mathbf{w}$ and $\mathbf{x}$:

$$f(\mathbf{w},\mathbf{x}) = h(\mathbf{w} \cdot \mathbf{x})$$

Then:

$$\begin{align*} f(\mathbf{w} + \delta \mathbf{w},\mathbf{x}) - f(\mathbf{w}, \mathbf{x}) & = h((\mathbf{w} + \delta \mathbf{w}) \cdot \mathbf{x}) - h(\mathbf{w} \cdot \mathbf{x}) \\ & \approx (\mathbf{x} \cdot \delta \mathbf{w})\, h'(\mathbf{w} \cdot \mathbf{x}) \end{align*}$$

Again, this gives you an approximation to how "wrong" the weights are given a "wrong" observation.

The backpropagation algorithm works by combining all of the training data into a single loss function which measures how "bad" the current set of weights is in training the data, and then does more or less exactly the above procedure, estimating the gradient of the loss function with respect to the weights, and then moving all of the weights in that direction.