3
$\begingroup$

Stochastic gradient descent with a batch size of 1 is apparently used to learn from single examples as they arrive, but I don't understand why you would use such a small batch size instead of batching training inputs together.

$\endgroup$
  • $\begingroup$ Your question seems like a non-sequitor to me. What makes you think that online learning only learns one weight at a time? Neither form of learning "learns only one weight at a time", as I understand them. I suspect possibly you might have a misconception. Please edit the question to clarify where you got the idea that one of these forms of learning "only learns one weight at a time", and what you think that has to do with the advantages of online learning vs batch learning. $\endgroup$ – D.W. Dec 30 '15 at 0:45
  • $\begingroup$ Edited for clarity, I refined my understanding of stochastic gradient descent and learning $\endgroup$ – bsm Dec 30 '15 at 4:29
6
$\begingroup$

Batch learning and online learning both have their place. Generally speaking batch learning will train your neural network to a lower residual error level, because the online training can sometimes have one training undo the effect of another. However, online training has a few advantages:

  • Online learning schemes learn "faster." In some cases, determining how many samples you need can be difficult, and sample gathering may be costly. Online learning lets you see the progress of your training as the number of samples increases, potentially saving on sample gathering costs once it reaches an acceptable error.
  • Online learning schemes can be done during operation, which can be valuable.
  • Online learning schemes are more effective for dealing with data sets that are not stationary. It is easier for the batch methods to come up with highly specialized solutions which work well for the test samples, but do not capture a general solution.
$\endgroup$
-1
$\begingroup$

One reason you'd use online learning is that the samples (data points) only arrive one at a time. If you have all of the samples (data points) available at once, then batch learning is at least an option: but if the samples arrive in a continuous, streaming fashion, one at a time, and you need to make the best decision you can at each point in time given only the samples you've seen so far, then online learning becomes attractive, because it provides a way to update the neural network as each new sample arrives.


Online learning does not mean "only learning one weight at a time". Online learning is about learning from one sample at a time (one data point at a time), but you're still updating all the weights of the neural network.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.