3
$\begingroup$

I am a CS student learning about neural nets. Currently I am confused about how to train a neural net in batches. If I calculate error in a batch, I will get a vector of errors e.g. real1 - predicted1, real2 - predicted2, etc... How do I then calculate a single gradient value to adjust the weights with?

$\endgroup$
3
$\begingroup$

This is known as stochastic gradient descent. I suggest you read some background material on stochastic gradient descent; then I think you will understand. There is lots written on that subject in many standard places.

Compare:

  • In standard gradient descent, we let $E(w)$ denote the total error across all training samples (as a function of the weights $w$), compute the gradient of $E(w)$ (with respect to $w$), and use that gradient to update the weights.

  • In stochastic gradient descent (in its simplest form), we pick a single training sample at random, let $E_1(w)$ compute the error on just that training sample (as a function of the weights $w$), compute the gradient of $E_1$ (with respect to $w$), and use that to update the weights. We repeat this many times. In each iteration, we pick a different randomly chosen training sample.

    $E_1(w)$ can be thought of as a noisy approximation to $E(w) $, so the gradient of $E_1(w)$ can be thought of as an approximation to the gradient of $E(w)$. Thus, in each iteration of stochastic gradient descent, we are moving in a direction that is roughly correlated with the direction standard gradient descent will take, but with some noise; over enough iterations, hopefully this noise will average out. A crude intuition might be that $kn$ iterations of stochastic gradient descent are roughly comparable to $k$ iterations of standard gradient descent, if $n$ is the number of samples in the training set.

  • Stochastic gradient descent with batches is a more sophisticated form of stochastic gradient descent, where we randomly choose a batch of training samples (more than one sample, but less than all of them), sum the total error across the batch, compute the gradient, and use that to update the weights.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.