Stochastic gradient descent with a batch size of 1 is apparently used to learn from single examples as they arrive, but I don't understand why you would use such a small batch size instead of batching training inputs together.
Batch learning and online learning both have their place. Generally speaking batch learning will train your neural network to a lower residual error level, because the online training can sometimes have one training undo the effect of another. However, online training has a few advantages:
- Online learning schemes learn "faster." In some cases, determining how many samples you need can be difficult, and sample gathering may be costly. Online learning lets you see the progress of your training as the number of samples increases, potentially saving on sample gathering costs once it reaches an acceptable error.
- Online learning schemes can be done during operation, which can be valuable.
- Online learning schemes are more effective for dealing with data sets that are not stationary. It is easier for the batch methods to come up with highly specialized solutions which work well for the test samples, but do not capture a general solution.
One reason you'd use online learning is that the samples (data points) only arrive one at a time. If you have all of the samples (data points) available at once, then batch learning is at least an option: but if the samples arrive in a continuous, streaming fashion, one at a time, and you need to make the best decision you can at each point in time given only the samples you've seen so far, then online learning becomes attractive, because it provides a way to update the neural network as each new sample arrives.
Online learning does not mean "only learning one weight at a time". Online learning is about learning from one sample at a time (one data point at a time), but you're still updating all the weights of the neural network.