Today Google's AI won its first game of Go against Lee Sedol, one of the best Go players on the planet. Image interpretation and self-driving cars are other recent success stories in machine learning. I was under the impression that these advances are largely the result of better algorithms and novel ideas, but some of the guys in my university were claiming these advances are mainly due to increased computational power. Neural networks in particular are an old concept that was largely forgotten for decades and now we're doing all sorts of cool new things with it.

What is the role of computational power with regards to recent successes in artificial intelligence?


1 Answer 1


This is a broad question, but from the presentations I've seen, there have been three major factors that all have come together over the past decade or two:

  • More data. Machine learning algorithms rely upon having many instances, so we can train them. The more data we have (the more labelled instances we have), the better the algorithms do. On some of the harder problems people are tackling today, machine learning algorithms only start to do well once you give them millions or billions of examples. For example, speech recognition is a prime example of this: researchers in the field have told me that progress in speech recognition was closely related to the size of the data sets available for training. Over the past decade or two, large data sets have become available: it's become possible to train algorithms using every image on Flickr, or every video on Youtube, or every call ever made to Google Voice. That wasn't possible two decades ago.

  • More computational power. Processing that much data requires efficient algorithms. One significant advance was that researchers found out how to use GPU's (graphic cards) as specialized processors: implementing the machine learning algorithm on a GPU could speed up the training process by 10x - 100x.

  • Better algorithms. Deep learning has been having some significant successes over the past five years or so. This is a result of decades of hard work. There's a small subcommunity of researchers who have been working on neural networks for decades, and who have slowly been making progress, finding what kinds of neural networks work well. They've invented many new algorithmic ideas: autoencoders, stochastic gradient descent with mini-batches, ReLU activation functions, drop-out for regularization, convolutational neural network structures.

It has taken the combination of all of these new algorithmic ideas, plus larger data sets, plus new ways of implementing the algorithms efficiently so we can make effective use of the large data sets, to achieve these gains. This is a real success story for the field.

Sometimes you hear the stereotype of a lone inventor sitting in their garage, who one day has a "Eureka!" moment and comes up with a brilliant invention fully-formed in a flash of insight. However, that's typically not how research progresses. Deep learning is a good example of that. It has taken decades of work and gradual improvement, exploring many ideas that ultimately didn't pan out, with researchers building on incremental advances and new ideas from other researchers, to reach where we are today. An inspiring example for the impact of long-term fundamental research.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.