4
$\begingroup$

This question is both general in nature and also specific to computer vision. If this is the wrong forum, apologies in advance and suggestions on where to post would be much appreciated.

After a certain point, do the benefits of more data plateau for machine learning algorithms?

For instance, let's say the goal is object recognition of a basketball. Is there a plateau, say, after training on 1M images of basketballs? Or is the plateau lower at like 100K images? Or is there no plateau at all?

More concretely, can you detect basketballs with 99% accuracy after 100K samples, meaning the next 900K samples only nets an additional 1% in accuracy at most?

How about for non-image domains such as speech recognition of all words related to the weather in the English language?

It seems that if there is a plateau, it would hinge on the complexity of the domain. Assuming the data plateau exists, is there a principle for generalizing what the plateau is for a given domain (e.g., to recognize one type of object with no variations, you need about 100K images from every angle and under different lighting conditions)?

$\endgroup$
  • $\begingroup$ Do you want to know how much the quality (e.g. accuracy) of your classifier will improve if you give it more training samples? $\endgroup$ – Martin Thoma Dec 1 '16 at 13:54
  • $\begingroup$ What exactly would you call a plateau? If the quality increases at some point linearly with log(log(# training samples)), is this a plateau? Is it only a plateau if it doesn't increase at all? If you can be certain not to reach such a platau for < 10^100, but not if ther might be one for 10^10000 training samples to reach such a plateau, would that be relevant at all? $\endgroup$ – Martin Thoma Dec 1 '16 at 13:57
  • $\begingroup$ @MartinThoma great, precise questions. :) hmm let's define "plateau" as when accuracy and speed reach 95% of human levels (i.e., algorithm detects basketball with 95% accuracy compared to humans and at 95% of speed)? $\endgroup$ – Crashalot Dec 1 '16 at 18:48
  • $\begingroup$ What does "algorithm detects basketball with 95% accuracy compared to humans" mean? Usually, you have a ground truth. On this data any classifier (no matter if human or algorithmic) can at most get 100%. Usually, humans are better than algorithms, but not always. For example, on ImageNet I've heard that humans are ~95% and algorithms are currently at ~97%. $\endgroup$ – Martin Thoma Dec 1 '16 at 19:27
  • $\begingroup$ @MartinThoma sorry for the confusion. okay, amend the question to ground truth so plateau is reached when accuracy is 95%. if this is the threshold, or even if 99% accuracy is the threshold, there is a point where more data is effectively meaningless, right? $\endgroup$ – Crashalot Dec 1 '16 at 19:40
4
$\begingroup$

Yes, typically there will be a plateau. There's usually no way to guess exactly where the plateau will be, a priori; the only way to find out is to build larger and larger data sets and see what happens.

The size of the data set needed to reach a "plateau" is dependent on many factors, including the specific classification task you're trying to solve, the machine learning algorithm you're using, the set of features you've chosen, and maybe other considerations. For some tasks, a fairly small training set might be sufficient to reach the "plateau". For others, you might need an incredibly enormous training set -- so large that you'll never reach it in practice, so effectively you'll never hit a plateau in practice.

There's absolutely no way to answer your specific questions about how many images are needed for particular domains. I know you're hoping there will be some rule of thumb that'll help you predict how many images you need for some new classification task, but I've got bad news for you; there's not really any useful rules of thumb that I'm aware of. The only way to find out is to try it, or to read research papers or experience reports from others who have tried to solve the same task or a similar task.

It's also worth keeping in mind that the notion of a plateau should not be taken too seriously. Rather, you should think of it as diminishing returns. After some point, increasing the size of the training set will give diminishing returns in accuracy. It's not to say that there's necessarily a point where you get literally zero improvement; it's just that the improvement available from a larger training set might be extremely small, i.e., where the accuracy curve starts to become nearly horizontal, when graphed as a function of the size of the training set.

In the comments you separately talk about "the point when a classifier reaches 95% of human accuracy". That's not the same thing as a plateau; that's a totally different measure. For some classification tasks, humans are way better than any classifier we know of; we don't know of any machine learning approach that will get anywhere near human accuracy, let alone 95% of human accuracy, no matter how large a training set you have. For other classification tasks, machine learning can do better than any human -- so the classifier gets not just 95% of human accuracy, but maybe 200% of human accuracy. Comparing a classifier's accuracy to human accuracy is unrelated to the existence and location of a plateau (i.e., diminishing returns as you increase the size of the training set).

$\endgroup$
  • $\begingroup$ Terrific answer, thanks so much! Confirmed my hunch. One reason for asking was to see if any rules of thumbs exist, but also to verify the media hype about Google's "data advantage" in machine learning. This answer suggests any "data advantage" in machine learning depends heavily on the domain, and in some cases, any lead Google has may get marginalized with enough training and time to catch up. $\endgroup$ – Crashalot Dec 2 '16 at 1:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.