I'm reading this chapter https://www.deeplearningbook.org/contents/mlp.html of the Deep Learning book, and on page 209, they have this equation (assume there is no regularizer and no bias parameter): $$ \nabla_{W^{(k)}}J = gh^{(k-1)T} $$ I'm not sure how this equation is derived, because I thought based on the equation they give on page 203: $\nabla_xz = \left(\frac{\partial y}{\partial x}\right)^T\nabla_y z$, then should we have $\nabla_{W^{(k)}}J = \left(\frac{\partial a^{(k)}}{\partial W^{(k)}}\right)^T\nabla_{a^{(k)}} J = h^{(k-1)T}g$ (because $g=\nabla_{a^{(k)}}$ and $a^{(k)} = W^{(k)}h^{k-1}$) instead?

  • $\begingroup$ Check the dimensions of the two formulas – you're giving a scalar, whereas the book is giving a matrix. $\endgroup$ – Yuval Filmus Apr 4 '20 at 7:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.