1
$\begingroup$

I am attempting to learn neural networks using the Keras libary on the MNIST hand written digit dataset, using dense layers only. I am trying to figure out what the best shape for the network should be but I can't seem to find any literature or discussion on the subject.

By shape, I mean should the hidden layers have more nodes as I go deeper, less nodes, or the same? I've taken to calling them square, pyramid, and reverse-pyramid because I couldn't find a better name. See the end of this post for clarification on shape

I've tried all 3 and the reverse pyramid is giving me the best at 80% accuracy but I can't imagine why. I feel like I would just be losing data and introducing noise by expanding the number of neurons as I go deeper.

Is there any literature or discussions about this subject? Perhaps, I'm not using the right keywords to look, I just don't know the correct terminology. I've tried neural network shape, structure, hidden layer layout, etc. with very little luck.

Thanks in advance.

Square - Hidden layers stay same size until the end

http://neuralnetworksanddeeplearning.com/images/tikz10.png

Pyramid - less neurons at each additonial hidden layer (pretend they are all hidden layers)

enter image description here

Reverse Pyramid - more neurons at each additiona hidden layer

Reverse Pyramid

Improve this question
$\endgroup$
2
  • $\begingroup$ I've seen such networks called narrow and wide in addition to being deep (multiple layers). To my best knowledge, nobody really knows what structure works best for a given problem. Take a small sample of your data and systematically try many choices. $\endgroup$
    – Juho
    Commented Feb 20, 2018 at 22:08
  • $\begingroup$ It sounds as if you were learning by doing only, which has its limits for topics like this. "Is there any literature or discussions about this subject?" -- obviously! There are many textbooks and courses on AI in general, and NNs in particular. I suggest you pick up some material of that kind. $\endgroup$
    – Raphael
    Commented Mar 23, 2018 at 5:59

2 Answers 2

Reset to default
1
$\begingroup$

Are you using the Training set for testing ? What is your train test split ?

With Neural Nets, it's always the pyramid shape that works. This is because as you go deeper, data about the input is condensed and we get a proper representation of the input for outputting the required class or regression value.

The only reason reverse pyramid nets will work well is overfitting, which accommodates noise and does not generalize well.

http://www.deeplearningbook.org/contents/mlp.html

Try the Bible of Deep Learning for more details.

Improve this answer
$\endgroup$
0
$\begingroup$

The most common case I've seen is that the number of neurons at each hidden layer decreases as you get closer to the output. However, there are no hard-and-fast rules. It may require some experimentation to see what works best in your particular application.

When working with images, convolutional neural networks (CNNs) often perform better than fully connected neural networks.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for your response. I seem to be able to get similar results with a square shape as well, but the loss graph and accuracy graph shows over-training. I'd still like to read more about this subject, as experimenting with different shapes is about all I've been able to do so far. I've still only gotten middling accuracy of around 80% though. I was hoping someone smarter than me has investigated what the properties of each shape would contribute the network's learning. $\endgroup$
    – Tryer
    Commented Feb 20, 2018 at 22:17
  • $\begingroup$ @Tryer, you should be able to do better than 80% accuracy. Do a websearch and you'll find a bunch of tutorials that describe examples of how to get high accuracy (e.g., ~ 98%) on MNIST with a few fully connected layers. They describe their architecture and how many neurons they use at each layer. $\endgroup$
    – D.W. ♦
    Commented Feb 20, 2018 at 23:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.