3
$\begingroup$

I am having a problem with my program of deep neural network using Theano. In my deep neural network, I have several layers of neural network to predict an output given a certain input. Because of an issue when compiling theano, I have to debug my program step-by-step. Since Theano gives me an error when updating weights of some layers, I perform a check by calculating the derivatives of only the last layer (using tensor.grad if you know Theano). But if I do that then the network will not update properly and the cost does not decrease and the output is nowhere close to the true output. It seems like updating the weights of only the last layer does not seem to train the network to learn.

Is it mandatory to update all weights of all layers in a deep neural network to decrease the cost? What I understand is that if I fix the values of all weights except the weights of the last layer, it should still be able to predict correctly after many rounds of training (of course, the only layer that is learning is the last layer, but I accept that, I only needs the predicted values to be close to my true values while training).

I hope my explanation make sense to you. If you do not understand any part, I will gladly provide you more information.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

If you're only changing the weights in the last layer, then effectively you have a neural network with a single layer, preceded by some preprocessing step. Single-layer neural networks (also known as perceptrons) aren't so powerful.

Why isn't the network learning anything? It doesn't learn much because it can't -- a single-layer network can't learn complicated functions.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Yeah, that's what I thought. But for some reasons, my neural network with single layer just does not want to learn anything. Very strange! Perhaps something wrong in code, not in the model $\endgroup$
    – The Lazy Log
    Commented Aug 3, 2016 at 15:22
  • $\begingroup$ @TheLazyLog, exactly -- it doesn't learn anything because it can't -- a single-layer network can't learn much. $\endgroup$
    – D.W. ♦
    Commented Aug 3, 2016 at 15:34
  • $\begingroup$ But I mean it should at least learn to overfit the single training example, right? The problem in my network is that it just does not learn at all. I just want it to fit exactly to the training example. $\endgroup$
    – The Lazy Log
    Commented Aug 4, 2016 at 1:20
  • $\begingroup$ @TheLazyLog Perceptrons cannot represent all functions. $\endgroup$
    – Yuval Filmus
    Commented Aug 4, 2016 at 5:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.