# What is the Effect of Hidden Layer Size?

I aim to gain more intuition with regards to the role of a hidden layer in a neural network architecture. And further to understand whether my classification problem is a linear one or not.

Here is what I have: I have a input of size 128 which is the output of the FFT of a multi-tone signal. Bellow you can find one input sample (vector size of 128 features): In my dataset, for each input sample the peaks are randomly located across 128 frequencies 90-127Hz].

The output is simply the number of the peaks in the input samples (FFT). f0r example the output/label for the provided sample in the figure is 8. I have total of 10 labels/classes, ranges from 1 to 10.

My data-set consists of 1000 samples/rows.

I choose the fully connected feedforward network with one hidden layer to be training using my datasest. thats is my network has 128 (+ 1 bios) units in the input layer, 1 hidden layer and the output layer with 10 unit in it.

To understand the effect of the the hidden layer in training and testing, I set up a simulation where I train my network 100 times per number of units in the hidden layer. The number of units in the hidden layer ranges from 0 (no hidden layer) to 40 units.

I used neural network toolbox in MATLAB to program this. Moreover, I utilized patternet command and used the default setting for configuration of network: the training function is traincg, the loss function to minimize error is cross-entropy loss, and I use softmax fuction for the output layer.

I configure the network, as well as initialize the weights before training.

And finally, for the training the network, I partition my data-set to 70/15/15 ratio corresponding to train/validation/test data blocks.

To gain a better insight, I plotted the misclassification error of network vs. the number of units in the hidden layer on the testing data block, which you can find below: now here are my questions:

a) Given the fact that I have the best performance when the network does not have a hidden layer, --is my classification problem a linear or a non-linear? --Are my classes linearly separable or not? (consider the high dimension of my input, 128)

b) Referring to the Fig. 2, --Why does the network using one hidden layer with 10 units (or around 10) produce the "second" most optimum performance among? --is it because I have 10 classes? --Moreover, if I have only two classes, would the network with one hidden layer size two get the better classification accuracy, or we can't answer this?

c) what is the role of the hidden layer?

Given the proven power of such technique in many complex real world problems, utilizing the neural network for this problem is over-kill. As the main goals is to have deep intuition on the neural network power in classification, I choose this toy problem.

I would appreciate if someone could answer above questions to help me have better insight regarding choosing the suitable network architecture. I can provide more details in case more clarification is needed.

• a) That's not how you tell whether a problem is linearly separable. b) You can probably answer your own question by doing another experiment where do the same experiment, but with 5 classes or 20 classes instead of 10. c) What do you mean by "what is the role of the hidden layer?" – D.W. Feb 20 '17 at 23:56
• a) I ran regression analysis. Studying the regression plot, I observe R = 0.9989 which I assume show that my data is linearly separable. Is this right? b) Thanks. c) I know that hidden layer introduce some sort of non-linearity to the network behavior.Is it true that by implementing an hidden layer or more, one can classify almost all complex patter recognition problem? If yes, how could one knows how many hidden layer to use in the network architecture? – tafteh Feb 21 '17 at 13:13
• In regard to c and your comment @tafteh , it has been proved in the past that one hidden layer is enough (Without restricting the number of neurons in that layer) to manage everything a multilayer nn would do (Much like a Single-track Turing Machine has the same "power" as a Multitrack Turing Machine ). The amount of hidden layers is sometime part of the problem and there is no right answer, and very much differs with the problem itself. In regard to A, an easy way to know if your problem is Linearly separable is to run a Hard-SVM and see if it fails or not (If it fails - not separable) – Zionsof Feb 21 '17 at 14:42
• Thanks @Zionsof for your answer. I'm trying to implement skilearn SVC(kernel='linear') in python. However, how do you mean by "if HARD-SVN fails?" since I can not visualize decision surface, what should I obtain to interpret if it is failing or not? – tafteh Feb 21 '17 at 16:51
• Try to run Hard-SVM on your data frame. If it fails (It will tell you) - then it is not linearly separable. That's because Hard-SVM has a strong assumption that the data is linearly separable to work, and does not work at all if even one sample makes it not linearly separable. – Zionsof Feb 26 '17 at 7:27