What is the difference between a Neural Network, a Deep Learning System and a Deep Belief Network?

As I recall your basic neural network is a 3 layers kinda thing, and I have had Deep Belief Systems described as being neural networks stacked on top of each other.

I've not til recently heard of a Deep Learning Systems, but i strongly suspect it is a synonym for Deep Belief System. Can anyone confirm this?

  • $\begingroup$ maybe you mean "deep learning"? see eg deep learning news/links $\endgroup$
    – vzn
    Commented Oct 29, 2013 at 15:34
  • $\begingroup$ Deep Belief System, is the term I ran into, they may or may not be synonyms (A google search will throw up articles for Deep Belief System) $\endgroup$ Commented Oct 29, 2013 at 23:03
  • $\begingroup$ Deep Belief Network is the canonical name because they derive from Deep Boltzmann Network (and it can be confusing with a belief propagation system which is totally different since it's about bayesian networks and probabilistic decision theory). $\endgroup$
    – gaborous
    Commented Jul 17, 2015 at 9:17
  • $\begingroup$ @gaborous Deep Belief Network is the correct name (the document I got years back introducing me to them must have had a typo). but as to being derived from deep boltzman networks, that name itself is noncanonical (AFAIK, happy to see a citation). DBNs derive from Sigmoid Belief Networks and stacked RBMs. I don't think the term Deep Boltzmann Network is used ever. On the other hand Deep Boltzmann Machine is a used term, but Deep Boltzmann Machines were created after Deep Belief Networks $\endgroup$ Commented Jul 17, 2015 at 11:05
  • $\begingroup$ @Oxinabox You're right, I've made a typo, it's Deep Boltzmann Machines, although it really ought to be called Deep Boltzmann Network (but then the acronym would be the same, so maybe that's why). I don't know which deep architecture was invented first, but Boltzmann machines are prior to semi-restricted bm. DBN and DBM are really the same construction except that the base network used as a repeating layer is a SRBM vs BM. $\endgroup$
    – gaborous
    Commented Jul 17, 2015 at 12:31

4 Answers 4


artificial neural networks models were generally restricted to only a few layers, say 3, for decades, for various reasons, including a math proof named Kolmogorovs thm that indicated they could theoretically approximate arbitrary functions with low error (but only with many neurons).

multilayer networks beyond that were not feasible/effective via prior learning techniques. a similar breakthrough came in 2006 by 3 different researchers Hinton, Le Cun, Bengio who were able to train much more layers. the prominent problem in this area was handwriting recognition.

the term "deep belief network" seems to originate with a 2006 paper by Hinton referring to Bayesian networks, which have close conceptual/theoretical connections/analogies with neural networks. "A fast learning algorith for deep belief nets"

see slide 17 in particular of this presentation deep neural networks

so the deep learning field is only ~½-decade old and is undergoing rapid research and development. Google, Facebook, Yahoo have all anounced deep-learning based initiatives and R&D is ongoing.

  • $\begingroup$ web site dedicated to deep learning by researchers in the field $\endgroup$
    – vzn
    Commented Oct 31, 2013 at 15:48

Artificial neural networks is a class of algorithms that include a lot of different kinds of algorithms based on graphs, so I won't detail here beyond what you asked because there's too much to say, since there are so many kinds of ANNs.

The first kind of artificial neural networks, the famous McCulloch-Pitts neurons, were linear, meaning that they could only solve linear decision problems (ie, datasets that could be linearly separable by drawing a line). Over time, this linear neural network model became known as the Perceptron or Adaline (depending on how you compute weights update).

Linear neural networks are just composed of a bipartite graph, where the left side nodes are the inputs and the right side nodes the output. Only the weights of the edges between these nodes are learnt (the nodes activation threshold can also be adjusted but this is rarely done).

Basic neural network such as Adaline or Perceptron (no hidden layer) Linear decision boundary

A big step was taken when shallow neural networks were invented: instead of having only a bipartite graph, we use a 3partite graph: the input "layer", the output "layer", and one "hidden layer" between them. Thank's to the hidden layer, the network can now take non-linear decisions, and solve problems such as the canonical XOR.

Xor neural network Xor neural network example activations paths Xor network decision boundary

Note that the "shallow" term was coined retrospectively when deep neural network (also called n-layers neural networks) were invented. This is to oppose neural networks with only one hidden layer, with deep neural networks with n hidden layers. As you can guess, using more hidden layers allow to decide on more complex datasets since there are more layers to modulate the decision (ie, in other words you're increasing the dimensionality of your decision boundary, which can lead to overfitting).

N-layer neural network Complex non-linear decision boundary using a n-layer deep neural network

You may ask: why did no-one try to use multi-layered (deep) neural networks before? In fact, they did, as early as 1975 by Fukushima with the Cognitron and Neocognitron (which is in fact a convolutional neural network, but that's another story). However, the issue was that noone knew how to efficiently learn such networks, the big issue being the regularization. Hinton's AutoEncoders opened the way, and later the Rectified Linear Units of LeCun fixed the issue for good.

What about Deep Beliefs Networks (DBN)? They are just multi-layered semi-restricted Boltzmann machines. So they are a kind of deep neural network, but with a different basic network pattern (ie: the layer, this is the pattern that is repeated): Boltzmann machines are different from other networks in that they are generative, meaning that they are usually used to learn from your data to reproduce it ("generate" it), while usual deep neural networks are used to separate your data (by drawing a "decision boundary").

In other words, DNN are great to classify/predict a value from your dataset, while DBN are great to "repair" a corrupted data (when I say repair, it's not only from corrupted data, it can also be on a perfectly fine data that you just want to fix a little to be more stereotypical as to more easily recognize with another neural network, such as hand-written digits).

In fact, to neatly summarize, you can say that AutoEncoders are a simpler form of Deep Belief Network. Here is an example of a DBN trained to recognize faces but NOT numbers, the numbers are automatically faded away (this is the "fixing" effect of DBN):

Deep Belief Network example on face recognition

So in the end, DBN and DNN are not opposite: they are complementary. For example, you can imagine a system to recognize hand-written characters that will first feed a character's image to a DBN to make it more stereotypical and then feed the stereotyped image to a DNN that will then output which character the image represents.

A last note: Deep Belief Nets are very close to Deep Boltzmann Machines: Deep Boltzmann Machines use layers of Boltzmann Machines (which are bidirectional neural networks, also called recurrent neural networks), while Deep Belief Nets use semi-restricted Boltzmann Machines (semi-restricted means that they are changed to unidirectional, thus it allows to use backpropagation to learn the network which is way more efficient than learning a reccurent network). Both networks are used for the same purpose (regenerating the dataset), but the computational cost is different (Deep Boltzmann Machines are significantly more costly to learn because of its recurrent nature: it is harder to "stabilize" the weights).

Bonus: about Convolutional Neural Networks (CNN), you will find a lot of contradictory and confusing claims, and usually you'll find that they are just deep neural networks. However, it seems the consensus is to use the original definition of Fukushima's Neocognitron: a CNN is a DNN that is forced to extract different features at different hierarchical levels by imposing a convolution before the activation (which a DNN can naturally do, but forcing it by setting a different convolution/activation function on different layers of the network may yield better result, at least that's the bet of CNN):

Deep neural network's features Convolutional neural network's features

And finally, for a more rigorous timeline of Artificial Intelligence, see here.

  • 1
    $\begingroup$ Great answer! A key aspect (perhaps the defining aspect) of convolutional deep networks is that each layer corresponds to applying a convolution then applying an activation function: there's a kernel, and all nodes in a layer apply the same kernel. Imagine the input is an image. Typically, you have a node for each pixel, and it has connections coming in from nearby pixels. An ordinary network would allow each pixel to have its own pattern of weights on the incoming edges. A convolutional network imposes the extra requirement that it's the same sequence of weights at every pixel in the image. $\endgroup$
    – D.W.
    Commented Jul 18, 2015 at 1:17
  • $\begingroup$ Yes indeed you're right, and that's where the name come from. But personally, I think this is often used as hack to specialize a deep neural network than anything else (when the defining characteristic of a deep neural network is to be general purpose and agnostic to the dataset). For example, HMAX, a kind of CNN, uses the convolution to compute saliency maps, which is clearly a hack when the saliency map in a biological neural network is of course not computed using dynamic programming... $\endgroup$
    – gaborous
    Commented Jul 18, 2015 at 9:00

Deep Neural Networks are Neural Networks which have relatively high depth. They are a subclass of Neural Networks. The idea actually goes back decades and is not new. What is new is that we have figured out how to train them in practice. The reason it has become very popular recently is training them became feasible and people used them to beat state of art algorithms. DNN need a lot of data and a lot of computational power which were not available decades ago. Geoff Hinton and his students and colleagues figured out how to train them in practice over the past decade and used them to beat the state of art machine learning algorithms in several fields where most researchers shunned the idea of using them completely at the time.

Belief Networks are a class of Probabilistic Graphical Models, they model a bunch of random variables and their dependencies using a graph (there are various ways to do this). Those variables themselves can be from parameterized distributions and might be modeled by neural networks (or some other model for representing parameterized distributions). Deep Belief Networks are Belief Networks which have relatively high depth.

Belief Networks tend to be generative, i.e. we can use a trained model to generate samples from the distribution it represents.

Neural networks tend to be discriminitive, we can use them to compute the probability of a given input (is this a picture of a cat? what is the probability of this being a picture of a cat?) but usually not for sampling (generate a picture of a cat).

  • $\begingroup$ I am not an expert on ML so take what I have written with a grain of salt. $\endgroup$
    – Kaveh
    Commented Jun 1, 2016 at 12:54

Neural networks are normally implemented where hidden layers and quadratic equations are required.where as deep learning and deep belief networks are utilized where multiple hidden layers are required for manipulation of data just like deep Boltzmann networks.

  • $\begingroup$ So both do require hidden layers? Could you do multiple hidden layers in normal NN? This Boltzmann networks, did you mean Boltzmann machines? So what is the difference? It does not answer the question (or maybe it does, but it is too unclear). $\endgroup$
    – Evil
    Commented Jun 1, 2016 at 4:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.