I am currently working my way through Michael Nielsen's ebook Neural Networks and Deep Learning and I am reading about overfitting and (L2) regularization. In this subsection, the process of L2 (a.k.a weight decay) regularization is introduced as being the process of adding the term $$ \frac{\lambda}{2n} \sum w^2 $$ to the cost function so that it becomes $$ C = C_0 + \frac{\lambda}{2n} \sum w^2 $$ where $C_0$ is the original cost function, $\lambda$ is the regularization parameter, $n$ is the number of items of training data being used, and the sum is over all of the weights in the network.

Later in this chapter, the process of choosing hyperparameters is discussed and the following is said:

In the above example [when we were using a training set of size 10,000] I left $\lambda$ as $\lambda = 1000.0$, as we used earlier. But since we changed the number of training examples [to 1,000] we should really change $\lambda$ to keep the weight decay the same. That means changing $\lambda$ to $20.0$.

Why would changing the number of training examples mean that we should change the value of the regularization parameter $\lambda$?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.