How large can the number of layers in a neural network be? Can there be maybe 1000 layers with, say, 1000 neurons in each layer?
I imagine the brain has hundreds of thousands of layers.
1
-
$\begingroup$ there are real NNs vs A(rtificial)NNs. this is a new area of study with deep networks. even then though the largest constructed nets are around a dozen layers, trained on supercomputers. the issue becomes that backpropagation tends to fail in effectiveness after "not that many" layers. new training algorithms/ theory are being sought. more in Computer Science Chat $\endgroup$– vznCommented Dec 13, 2015 at 19:39
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
In principle, there is no limit on the number of hidden layers that can be used in an artificial neural network. Such networks can be trained using "stacking" or other techniques from the deep learning literature. Yes, you could have 1000 layers, though I don't know if you'd get much benefit: in deep learning I've more typically seen somewhere between 1-20 hidden layers, rather than 1000 hidden layers. In practice the number of layers is based upon pragmatic concerns, e.g., what will lead to good accuracy with reasonable training time and without overfitting.
Don't confuse artificial neural networks with the brain. The behavior of artificial neural networks doesn't necessarily tell us anything about the brain, and you can build artificial neural networks whose structure doesn't match anything found in the brain.
$\begingroup$
$\endgroup$
The Universal Approximation Theorem says that a 3-layer network actually suffices to approximate any continuous function on $\mathbb{R}^n$.
$\begingroup$
$\endgroup$
To expand on @D.W.♦'s answer.
They are correct in stating
In principle, there is no limit on the number of hidden layers that can be used in an artificial neural network
There are however multiple challenges when training a very deep neural network. Challenges include the infamous vanishing/exploding gradient, local minima, overfitting, and time/memory restrictions.
Recent advancements have worked to solve these challenges.
- Distributed GPU Computation
- Fractal Neural Nets capable of being 100 layers deep without any use of residual tech
- Residual Neural Nets and Recurrent Neural Nets both capable of being 100-1000 layers deep
While depth seems to improve performance within neural net problems it has also been shown that width plays an important factor as well.
It is important to understand Neural Nets are a relatively new technology and that we are still in the process of learning about them.
Now to address your comment about the brain,
I imagine the brain has hundreds of thousands of layers.
Like @D.W.♦ mentioned, be careful not to confuse artificial neural networks with the human brain. The two are very different. That being said, researchers are working on making artificial networks like biological networks.
