I aim to gain more intuition with regards to the role of a hidden layer in a neural network architecture. And further to understand whether my classification problem is a linear one or not.
Here is what I have: I have a input of size 128 which is the output of the FFT of a multi-tone signal. Bellow you can find one input sample (vector size of 128 features):
In my dataset, for each input sample the peaks are randomly located across 128 frequencies 90-127Hz].
The output is simply the number of the peaks in the input samples (FFT). f0r example the output/label for the provided sample in the figure is 8. I have total of 10 labels/classes, ranges from 1 to 10.
My data-set consists of 1000 samples/rows.
I choose the fully connected feedforward network with one hidden layer to be training using my datasest. thats is my network has 128 (+ 1 bios) units in the input layer, 1 hidden layer and the output layer with 10 unit in it.
To understand the effect of the the hidden layer in training and testing, I set up a simulation where I train my network 100 times per number of units in the hidden layer. The number of units in the hidden layer ranges from 0 (no hidden layer) to 40 units.
I used neural network toolbox in MATLAB to program this. Moreover, I utilized patternet command and used the default setting for configuration of network: the training function is traincg, the loss function to minimize error is cross-entropy loss, and I use softmax fuction for the output layer.
I configure the network, as well as initialize the weights before training.
And finally, for the training the network, I partition my data-set to 70/15/15 ratio corresponding to train/validation/test data blocks.
To gain a better insight, I plotted the misclassification error of network vs. the number of units in the hidden layer on the testing data block, which you can find below:
now here are my questions:
a) Given the fact that I have the best performance when the network does not have a hidden layer, --is my classification problem a linear or a non-linear? --Are my classes linearly separable or not? (consider the high dimension of my input, 128)
b) Referring to the Fig. 2, --Why does the network using one hidden layer with 10 units (or around 10) produce the "second" most optimum performance among? --is it because I have 10 classes? --Moreover, if I have only two classes, would the network with one hidden layer size two get the better classification accuracy, or we can't answer this?
c) what is the role of the hidden layer?
Given the proven power of such technique in many complex real world problems, utilizing the neural network for this problem is over-kill. As the main goals is to have deep intuition on the neural network power in classification, I choose this toy problem.
I would appreciate if someone could answer above questions to help me have better insight regarding choosing the suitable network architecture. I can provide more details in case more clarification is needed.