# Proving Monotonicity of Softmax Layer

In the book here: http://neuralnetworksanddeeplearning.com/chap3.html

If you scroll down to Exercise 2 in the Softmax Section, it says

Show that $$\partial a^L_{j}/\partial z^L_{k}$$ is positive if $$j=k$$ and negative if $$j \neq k$$. As a consequence, increasing $$z^L_j$$ is guaranteed to increase the corresponding output activation, $$a^L_j$$, and will decrease all the other output activations.

Here, $$a_j = \frac{e^{z^L_{j}}}{\sum_{k}{e^{z^L_{k}}}}$$

I managed to prove the part when $$j \neq k$$ by differentiating as normal to get $$-\frac{e^{z^L_{j}}}{\left(\sum_{k}{e^{z^L_{k}}}\right)^2}$$

which is obviously always negative. However I'm having trouble with when $$j=k$$. When I differentiated I got an inequality which simplified to proving $$\sum_{k}{e^{z^L_k}}>1$$ I am unsure of how to do this.

Let's remove the $$L$$ superscripts. The derivative with respect to $$z_L$$ is $$\frac{\partial a_j}{\partial z_j} = \frac{e^{z_j} \sum_k e^{z_k} - e^{z_j} e^{z_j}}{\left(\sum_k e^{z_k}\right)^2} = \frac{e^{z_j} \sum_{k \neq j} e^{z_k}}{\left(\sum_k e^{z_k}\right)^2} > 0.$$