I have been implementing some neural networks in MATLAB and recently I noticed a weird thing while implementing softmax derivative:

Setting the derivative to one, rather than using the actual derivative converges faster.

I have to admit that the derivative of softmax in particular confused me quite a bit, since the actual derivative requires the Jacobian as opposed to other activation functions that only depend on the input. That is, for output vector of size nx1, the derivative of the activation function is also nx1 (see ReLU, tanh etc.) but for softmax is nxn.

Anyway, the simple case is to consider the diagonal of the Jacobian which is “out*(1-out)” as derived and proposed in a couple of sources around [1-3]. However setting the derivative to “1” shows faster convergence.

enter image description here

I am not fully sure to say how this phenomenon generalises but I tried with different settings, architectures and optimisers and the issue seems to persist.

I made an example of a 2x5x5x2 feedforward network with tanh in all but last layer, where I’ve used softmax, classifying a typical XOR problem using cross-entropy loss with ADAM optimizer.

Does anyone have any hunch why this would happen?


1: https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/
2: https://math.stackexchange.com/questions/945871/derivative-of-softmax-loss-function
3: https://datascience.stackexchange.com/questions/29735/how-to-apply-the-gradient-of-softmax-in-backprop

P.S. I am aware that showing this behaviour in only one problem does not mean that it happens always, but in a couple of experiments I have done, this behaviour seems to be persistent


I'm not sure what you mean by "the simple case is to consider the diagonal of the Jacobian". Are you perhaps doing some unusual variation of gradient descent where you compute only the diagonal of the Jacobian, but not the other entries? If you are, then don't be surprised if it performs poorly. Don't do that. I wouldn't expect that to work well.

Instead, I recommend you follow the standard approach: use backpropagation to compute the gradient and use gradient descent to update the weights. There are many standard tutorials on how to do that. Realistically, we never need to compute the Jacobian matrix for the softmax directly. Instead, we look at the output of the loss function, and we compute the gradient of that with respect to the weights. That doesn't require a Jacobian, because we are computing the gradient of a function that has a single output. Then, we use the gradient to update the weights (that's the gradient descent part).

If you're implementing this yourself from scratch, there are many things that can go wrong. I recommend implementing a gradient check to make sure you are computing the gradient properly. But, more realistically, I recommend using some existing framework for training neural networks (e.g., Keras, Torch, etc.) rather than trying to implement it from scratch. If you implement from scratch, there are many ways you can create some small implementation error that will lead to results that are subpar but hard to debug.

  • $\begingroup$ Hey, thanks for your input. Indeed I implement it from scratch for a variety of reasons. I need to make some very modular implementations for some research purposes but no need to get there now. So, I indeed use gradient descent in back propagation through the chain rule to update the weights: dE/dw = dE/dout * dout/din * din/dw , etc. So the second part dout/din is what I am talking about, which is straightforward for tanh etc. (continues in next comment) $\endgroup$ Aug 17 '18 at 7:52
  • $\begingroup$ (continues from previous comment) In softmax, each output depends also on the input of the other outputs, since softmax performs normalization. That forms a Jacobian. If I take only the current output with regard to its input, then this is the diagonal of the Jacobian. Which doesn’t work quite well as shown in the figure above I remember myself at some point doing gradient checks but indeed haven’t tested it in this case. Numerical gradient would be dead slow but should give at least a hint what’s going on. Thanks again $\endgroup$ Aug 17 '18 at 7:56
  • $\begingroup$ @A.Gotsopoulos, when you say "If I take only...", you're not computing the gradient; you're doing something else. Don't do that. It's not going to work. Instead, compute the gradient properly, and then everything will work nicely. It sounds like you need to ask a separate question about how to compute the gradient of the loss function applied to a softmax. You'll discover that the answer is not "first I fill in each entry of the Jacobian matrix of the softmax, then...". That works, but there is a faster way: use backpropagation. I can't explain the details in a comment. $\endgroup$
    – D.W.
    Aug 17 '18 at 16:02
  • $\begingroup$ I do compute the gradient. I do backpropagation. I calculate the gradient analytically and apply backpropagation using the chain rule. Maybe I am not expressing myself well but I do all of these. I did a gradient check, the numerical and analytical gradients agree yet I do not see a better behaviour than setting the derivative to 1 as I mentioned in the original post. Thanks for your time. $\endgroup$ Aug 18 '18 at 16:50
  • $\begingroup$ @A.Gotsopoulos, sorry, but I don't know how to reconcile your statements about taking only the diagonal of the Jacobian with your claims that you compute the gradient correctly. If you are considering only the diagonal entries of the Jacobian and ignoring the rest (or treating them as 0), then you aren't computing the gradient of the loss function correctly. So I don't know how to reconcile that with your other statements. Since I can't understand what you're actually doing, I guess I don't know how to help any more. I hope someone else can help you with whatever I'm missing! $\endgroup$
    – D.W.
    Aug 18 '18 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.