What is the difference between a Neural Network, a Deep Learning System and a Deep Belief Network?
As I recall your basic neural network is a 3 layers kinda thing, and I have had Deep Belief Systems described as being neural networks stacked on top of each other.
I've not til recently heard of a Deep Learning Systems, but i strongly suspect it is a synonym for Deep Belief System. Can anyone confirm this?
artificial neural networks models were generally restricted to only a few layers, say 3, for decades, for various reasons, including a math proof named Kolmogorovs thm that indicated they could theoretically approximate arbitrary functions with low error (but only with many neurons).
multilayer networks beyond that were not feasible/effective via prior learning techniques. a similar breakthrough came in 2006 by 3 different researchers Hinton, Le Cun, Bengio who were able to train much more layers. the prominent problem in this area was handwriting recognition.
the term "deep belief network" seems to originate with a 2006 paper by Hinton referring to Bayesian networks, which have close conceptual/theoretical connections/analogies with neural networks. "A fast learning algorith for deep belief nets"
see slide 17 in particular of this presentation deep neural networks
so the deep learning field is only ~½-decade old and is undergoing rapid research and development. Google, Facebook, Yahoo have all anounced deep-learning based initiatives and R&D is ongoing.
Artificial neural networks is a class of algorithms that include a lot of different kinds of algorithms based on graphs, so I won't detail here beyond what you asked because there's too much to say, since there are so many kinds of ANNs.
The first kind of artificial neural networks, the famous McCulloch-Pitts neurons, were linear, meaning that they could only solve linear decision problems (ie, datasets that could be linearly separable by drawing a line). Over time, this linear neural network model became known as the Perceptron or Adaline (depending on how you compute weights update).
Linear neural networks are just composed of a bipartite graph, where the left side nodes are the inputs and the right side nodes the output. Only the weights of the edges between these nodes are learnt (the nodes activation threshold can also be adjusted but this is rarely done).
A big step was taken when shallow neural networks were invented: instead of having only a bipartite graph, we use a 3partite graph: the input "layer", the output "layer", and one "hidden layer" between them. Thank's to the hidden layer, the network can now take non-linear decisions, and solve problems such as the canonical XOR.
Note that the "shallow" term was coined retrospectively when deep neural network (also called n-layers neural networks) were invented. This is to oppose neural networks with only one hidden layer, with deep neural networks with n hidden layers. As you can guess, using more hidden layers allow to decide on more complex datasets since there are more layers to modulate the decision (ie, in other words you're increasing the dimensionality of your decision boundary, which can lead to overfitting).
You may ask: why did no-one try to use multi-layered (deep) neural networks before? In fact, they did, as early as 1975 by Fukushima with the Cognitron and Neocognitron (which is in fact a convolutional neural network, but that's another story). However, the issue was that noone knew how to efficiently learn such networks, the big issue being the regularization. Hinton's AutoEncoders opened the way, and later the Rectified Linear Units of LeCun fixed the issue for good.
What about Deep Beliefs Networks (DBN)? They are just multi-layered semi-restricted Boltzmann machines. So they are a kind of deep neural network, but with a different basic network pattern (ie: the layer, this is the pattern that is repeated): Boltzmann machines are different from other networks in that they are generative, meaning that they are usually used to learn from your data to reproduce it ("generate" it), while usual deep neural networks are used to separate your data (by drawing a "decision boundary").
In other words, DNN are great to classify/predict a value from your dataset, while DBN are great to "repair" a corrupted data (when I say repair, it's not only from corrupted data, it can also be on a perfectly fine data that you just want to fix a little to be more stereotypical as to more easily recognize with another neural network, such as hand-written digits).
In fact, to neatly summarize, you can say that AutoEncoders are a simpler form of Deep Belief Network. Here is an example of a DBN trained to recognize faces but NOT numbers, the numbers are automatically faded away (this is the "fixing" effect of DBN):
So in the end, DBN and DNN are not opposite: they are complementary. For example, you can imagine a system to recognize hand-written characters that will first feed a character's image to a DBN to make it more stereotypical and then feed the stereotyped image to a DNN that will then output which character the image represents.
A last note: Deep Belief Nets are very close to Deep Boltzmann Machines: Deep Boltzmann Machines use layers of Boltzmann Machines (which are bidirectional neural networks, also called recurrent neural networks), while Deep Belief Nets use semi-restricted Boltzmann Machines (semi-restricted means that they are changed to unidirectional, thus it allows to use backpropagation to learn the network which is way more efficient than learning a reccurent network). Both networks are used for the same purpose (regenerating the dataset), but the computational cost is different (Deep Boltzmann Machines are significantly more costly to learn because of its recurrent nature: it is harder to "stabilize" the weights).
Bonus: about Convolutional Neural Networks (CNN), you will find a lot of contradictory and confusing claims, and usually you'll find that they are just deep neural networks. However, it seems the consensus is to use the original definition of Fukushima's Neocognitron: a CNN is a DNN that is forced to extract different features at different hierarchical levels by imposing a convolution before the activation (which a DNN can naturally do, but forcing it by setting a different convolution/activation function on different layers of the network may yield better result, at least that's the bet of CNN):
And finally, for a more rigorous timeline of Artificial Intelligence, see here.
Deep Neural Networks are Neural Networks which have relatively high depth. They are a subclass of Neural Networks. The idea actually goes back decades and is not new. What is new is that we have figured out how to train them in practice. The reason it has become very popular recently is training them became feasible and people used them to beat state of art algorithms. DNN need a lot of data and a lot of computational power which were not available decades ago. Geoff Hinton and his students and colleagues figured out how to train them in practice over the past decade and used them to beat the state of art machine learning algorithms in several fields where most researchers shunned the idea of using them completely at the time.
Belief Networks are a class of Probabilistic Graphical Models, they model a bunch of random variables and their dependencies using a graph (there are various ways to do this). Those variables themselves can be from parameterized distributions and might be modeled by neural networks (or some other model for representing parameterized distributions). Deep Belief Networks are Belief Networks which have relatively high depth.
Belief Networks tend to be generative, i.e. we can use a trained model to generate samples from the distribution it represents.
Neural networks tend to be discriminitive, we can use them to compute the probability of a given input (is this a picture of a cat? what is the probability of this being a picture of a cat?) but usually not for sampling (generate a picture of a cat).
Neural networks are normally implemented where hidden layers and quadratic equations are required.where as deep learning and deep belief networks are utilized where multiple hidden layers are required for manipulation of data just like deep Boltzmann networks.
