1
$\begingroup$

I have a fully connected neural network that consists of 3 inputs + bias, 4 neuron hidden layer, and 2 layer output, I am using the sigmoid activation function on the hidden layer only and use the backpropagation algorithm. I am using the following training data:

[ 0.50,  1.00, 0.75, 1],
[ 1.00,  0.50, 0.75, 1],
[ 1.00,  1.00, 1.00, 1],
[-0.01,  0.50, 0.25, 1],
[ 0.50, -0.25, 0.13, 1],
[ 0.01,  0.02, 0.05, 1]

with the following target outputs

[1, 0],
[1, 0],
[1, 0],
[0, 1],
[0, 1],
[0, 1]

I want to understand when I should be stopping the training of data, I've read a few scientific/academic articles mentioning not to overfit or underfit the number of iterations (epochs) you have. However, I am struggling to know when.

The graph I get after 1000 epochs showing the mean square error and root square error rates: enter image description here

If someone can tell me / give me some advice on the optimal amount of epochs for my neural network. Or if I could be linked some more academic or scientific articles talking about this issue, it would be great!

$\endgroup$

1 Answer 1

2
$\begingroup$

If you have already split your data into three sets (i.e. training, validation and testing), then it is useful to track the training loss and validation loss for each epoch. The number of epochs is like any other hyperparameter you have to tune. A classic situation of overfitting occurs when your training accuracy keeps increasing while your validation accuracy remains the same (or in some cases even starts falling). The point at which your validation accuracy stops showing improvement compared to your training accuracy is when you should stop.

$\endgroup$
2
  • $\begingroup$ I see so i don't have a validation data set so the only metrics I can track is the MSE, and probability distribution, is it possible to justify when to top training based on just these factors? $\endgroup$
    – sfhdfh23
    Mar 27 at 17:14
  • $\begingroup$ You should split your data into training, validation and testing sets. This is highly important to save a part of the data for testing and for validation $\endgroup$
    – nir shahar
    Mar 27 at 17:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.