Linear Regression using a Neural Network

Question

I am trying to create a regression model using a Neural Network. I am currently learning how to work with neural networks (deeplearning.ai) and so the model is not implemented using any existing frameworks like keras.

Based on what I have learned,

• the model is configured to use no activation function in the output layer (which, obviously, has only one node).
• Input is images of fashion articles (shirt, jackets etc).
• output predicts prices for input article images.
• Hidden layers all use ReLU.
• Random initialization is done for all weights.
• Cost function is Mean Squared Error: $$J = \frac{1}{2m}\sum_{i=1}^m (a^{(i)} - y^{(i)})^2$$

Based on the formula for backpropogation, the last layer should get the error based on cost function. For a single example, we have: $$ \mathcal L = \frac{1}{2}(a - y)^2 $$ $$ error = \frac{d\mathcal L}{da} = (a - y) $$

where L is the loss function, a is the predicted value and y is the actual price of the article.

For the entire training set, we have (vectorized): $$ error = \frac{d\mathcal L}{dA} = (A - Y) $$ where m is the number of examples, A, Y are (1 * m) vectors where each value corresponds to each single example. A contains m predictions and Y has all the m prices.

Is this error value correct?

My problem is that the model converges to a what appears to be a local minima. The error after a few 100 iterations gets stuck. The error is also not very small (~ 0.0005). I am not sure if I have the equations right.

Answer 1

No. This is not the correct way to train such a network. To update the network, we want to update the weights of the network. Thus instead of using $\frac{d\mathcal L}{dA}$ to update the weights, you should be using $\frac{d\mathcal L}{dW}$, where $W$ are the weights. This could be computed as

$$\frac{d\mathcal L}{dW} = \frac{d\mathcal L}{dA} \times \frac{d\mathcal A}{dW},$$

where $\frac{d\mathcal L}{dA}$ is computed using the formula you obtained ($\frac{d\mathcal L}{dA} = A-Y$ is correct), and where $\frac{d\mathcal A}{dW}$ is computed using backpropagation through the network.

Or, better yet, use an existing framework that performs automatic differentiation for you (e.g., Keiras, Tensorflow), and save yourself a lot of effort.