Linear Regression using a Neural Network

I am trying to create a regression model using a Neural Network. I am currently learning how to work with neural networks (deeplearning.ai) and so the model is not implemented using any existing frameworks like keras.

Based on what I have learned,

• the model is configured to use no activation function in the output layer (which, obviously, has only one node).
• Input is images of fashion articles (shirt, jackets etc).
• output predicts prices for input article images.
• Hidden layers all use ReLU.
• Random initialization is done for all weights.
• Cost function is Mean Squared Error: $$J = \frac{1}{2m}\sum_{i=1}^m (a^{(i)} - y^{(i)})^2$$

Based on the formula for backpropogation, the last layer should get the error based on cost function. For a single example, we have: $$\mathcal L = \frac{1}{2}(a - y)^2$$ $$error = \frac{d\mathcal L}{da} = (a - y)$$

where L is the loss function, a is the predicted value and y is the actual price of the article.

For the entire training set, we have (vectorized): $$error = \frac{d\mathcal L}{dA} = (A - Y)$$ where m is the number of examples, A, Y are (1 * m) vectors where each value corresponds to each single example. A contains m predictions and Y has all the m prices.

Is this error value correct?

My problem is that the model converges to a what appears to be a local minima. The error after a few 100 iterations gets stuck. The error is also not very small (~ 0.0005). I am not sure if I have the equations right.

• Note that this question may be better at home on either of Cross Validated, Computational Science, or Artificial Intelligence. Do you want us to migrate it? – Raphael Nov 21 '18 at 16:30
• @Raphael, if you think it'd be better suited then yes, please. That would be great. – abhink Nov 21 '18 at 17:32
• Do you really mean a linear regression model, or do you mean a regression model? Neural networks are non-linear (unless you limit them severely -- e.g., one layer, no activation function, etc., and at that point it's no longer reasonable to call it a neural network). – D.W. Nov 21 '18 at 18:53
• Please define all notation. What is $a$? What is $y$? – D.W. Nov 21 '18 at 18:53
• @D.W. I have added definitions. I do mean a regression model. From what I have learned, using something like a sigmoid function in the output layer would restrict the output to (0, 1). Not using any activation would prevent that and output continuous values. – abhink Nov 21 '18 at 19:41

No. This is not the correct way to train such a network. To update the network, we want to update the weights of the network. Thus instead of using $$\frac{d\mathcal L}{dA}$$ to update the weights, you should be using $$\frac{d\mathcal L}{dW}$$, where $$W$$ are the weights. This could be computed as
$$\frac{d\mathcal L}{dW} = \frac{d\mathcal L}{dA} \times \frac{d\mathcal A}{dW},$$
where $$\frac{d\mathcal L}{dA}$$ is computed using the formula you obtained ($$\frac{d\mathcal L}{dA} = A-Y$$ is correct), and where $$\frac{d\mathcal A}{dW}$$ is computed using backpropagation through the network.
• Thank you for the reply. I am using the above mentioned equation to update weights. My question was whether I was using the correct derivative for loss function w.r.t. A. If it is indeed correct then what would be the next thing to check? I am using about a 100 images for input and about a 1000 iterations. Do you think adding more data and increasing number of iterations would help? – abhink Nov 22 '18 at 14:01