I was reading this article here: https://towardsdatascience.com/how-does-back-propagation-in-artificial-neural-networks-work-c7cad873ea7.
When he gets to the part where he calculates the loss at every node, he says to use the following formula:
delta_0 = w . delta_1 . f'(z)
where values delta_0, w and f’(z) are those of the same unit’s, while delta_1 is the loss of the unit on the other side of the weighted link."
And $f$ is the activation function.
He then says:
"You can think of it this way, in order to get the loss of a node (e.g. Z0), we multiply the value of its corresponding f’(z) by the loss of the node it is connected to in the next layer (delta_1), by the weight of the link connecting both nodes."
However, he doesn't actually explain why we need the derivative term. Where does that term come from and why do we need it?
My idea so far is this:
The fact that the identity activation function causes the term to disappear is a hint. The node doesn't feed into the next exactly as is, it depends on the activation function. When the activation function is the identity, the loss at that node just passes to the next one based on the weight. Basically, you just need to factor in the activation function somehow, specifically in a way that doesn't matter when it's the identity, and of course the derivative is a way to do this.
The issue is that this isn't very rigorous, so I'm looking for a slightly more detailed explanation.