I am trying to create a regression model using a Neural Network. I am currently learning how to work with neural networks (deeplearning.ai) and so the model is not implemented using any existing frameworks like keras.

Based on what I have learned,

  • the model is configured to use no activation function in the output layer (which, obviously, has only one node).
  • Input is images of fashion articles (shirt, jackets etc).
  • output predicts prices for input article images.
  • Hidden layers all use ReLU.
  • Random initialization is done for all weights.
  • Cost function is Mean Squared Error: $$J = \frac{1}{2m}\sum_{i=1}^m (a^{(i)} - y^{(i)})^2$$

Based on the formula for backpropogation, the last layer should get the error based on cost function. For a single example, we have: $$ \mathcal L = \frac{1}{2}(a - y)^2 $$ $$ error = \frac{d\mathcal L}{da} = (a - y) $$

where L is the loss function, a is the predicted value and y is the actual price of the article.

For the entire training set, we have (vectorized): $$ error = \frac{d\mathcal L}{dA} = (A - Y) $$ where m is the number of examples, A, Y are (1 * m) vectors where each value corresponds to each single example. A contains m predictions and Y has all the m prices.

Is this error value correct?

My problem is that the model converges to a what appears to be a local minima. The error after a few 100 iterations gets stuck. The error is also not very small (~ 0.0005). I am not sure if I have the equations right.

  • $\begingroup$ Note that this question may be better at home on either of Cross Validated, Computational Science, or Artificial Intelligence. Do you want us to migrate it? $\endgroup$
    – Raphael
    Nov 21, 2018 at 16:30
  • $\begingroup$ @Raphael, if you think it'd be better suited then yes, please. That would be great. $\endgroup$
    – abhink
    Nov 21, 2018 at 17:32
  • $\begingroup$ Do you really mean a linear regression model, or do you mean a regression model? Neural networks are non-linear (unless you limit them severely -- e.g., one layer, no activation function, etc., and at that point it's no longer reasonable to call it a neural network). $\endgroup$
    – D.W.
    Nov 21, 2018 at 18:53
  • $\begingroup$ Please define all notation. What is $a$? What is $y$? $\endgroup$
    – D.W.
    Nov 21, 2018 at 18:53
  • $\begingroup$ @D.W. I have added definitions. I do mean a regression model. From what I have learned, using something like a sigmoid function in the output layer would restrict the output to (0, 1). Not using any activation would prevent that and output continuous values. $\endgroup$
    – abhink
    Nov 21, 2018 at 19:41

1 Answer 1


No. This is not the correct way to train such a network. To update the network, we want to update the weights of the network. Thus instead of using $\frac{d\mathcal L}{dA}$ to update the weights, you should be using $\frac{d\mathcal L}{dW}$, where $W$ are the weights. This could be computed as

$$\frac{d\mathcal L}{dW} = \frac{d\mathcal L}{dA} \times \frac{d\mathcal A}{dW},$$

where $\frac{d\mathcal L}{dA}$ is computed using the formula you obtained ($\frac{d\mathcal L}{dA} = A-Y$ is correct), and where $\frac{d\mathcal A}{dW}$ is computed using backpropagation through the network.

Or, better yet, use an existing framework that performs automatic differentiation for you (e.g., Keiras, Tensorflow), and save yourself a lot of effort.

  • $\begingroup$ Thank you for the reply. I am using the above mentioned equation to update weights. My question was whether I was using the correct derivative for loss function w.r.t. A. If it is indeed correct then what would be the next thing to check? I am using about a 100 images for input and about a 1000 iterations. Do you think adding more data and increasing number of iterations would help? $\endgroup$
    – abhink
    Nov 22, 2018 at 14:01
  • $\begingroup$ @abhink, sorry, I don't know. In my experience, open-ended debugging questions don't work well in our site format here. $\endgroup$
    – D.W.
    Nov 23, 2018 at 23:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.