1
$\begingroup$

A question from chapter 3 of Michael Nielsen's [Neural Networks and Deep Learning]:

It's tempting to use gradient descent to try to learn good values for hyper-parameters such as the regularization parameter $\lambda$ and the learning rate $\eta$. Can you think of an obstacle to using gradient descent to determine values of $\lambda$ and $\eta$?

What are the challenges of using gradient descent on hyper parameters λ and η to find out their optimum values?

enter image description here

C is the L2 regularized cost function.

n is the length of the training data

w is the weight

The Learning rule for the weights will be:

enter image description here

I am having trouble understanding how λ and η will affect the cost function. I know that we should decrease η as we get closer to the Cost function minima. I don't have a clue how to optimize λ. I think that the values of the hyper parameters will change for every epoch so there can't be constant optimal values.

Improve this question
$\endgroup$
2
  • $\begingroup$ I'm not sure I understand what you are asking. Do you mean, updating all three of $w,\lambda,\eta$ in each iteration of gradient descent? Or, do you mean doing nested gradient descent: in the inner loop, update $w$ until convergence; in an outer loop, update $\eta,\lambda$ a little then repeatedly apply the inner loop. Or something else? You show how you plan to update the weights $w$, but I'm not clear on how you plan to update $\lambda,\eta$. $\endgroup$
    – D.W. ♦
    Apr 26, 2017 at 17:32
  • $\begingroup$ @D.W.♦ I have edited the question and framed it more accurately.I wanted to know what difficulties one may face in using gradient descent to optimize hyper-parameters. You can think of the question "how should I plan to update λ,η " $\endgroup$
    – Kartik chincholikar
    Apr 26, 2017 at 19:26

2 Answers 2

Reset to default
4
$\begingroup$

The cost function $C(w,\lambda)$ is not a function of learning rate $\eta$. You can't compute the gradient of $\eta$.

$\lambda$ is part of the cost function. You can indeed compute its gradient and apply SGD on $\lambda$. However, when you do that, $\lambda$ will keep decreasing because its optimal value is obviously $-\infty$.

Note that it is possible to optimize hyperparameters like $\eta$ and $\lambda$ with another neural net, but that is a different story.

Improve this answer
$\endgroup$
3
  • $\begingroup$ It doesn't matter. As long as $C_0(w)$ is bounded and $w$ is non-zero, the minimal of $C(w,\lambda)$ is $-\infty$ when $\lambda=-\infty$. $\endgroup$
    – user172818
    Apr 26, 2017 at 22:26
  • $\begingroup$ In practice we normally have the constraint that $\lambda \ge 0$ (otherwise that term isn't serving as regularization any longer). So $\lambda = -\infty$ isn't an admissible solution. $\endgroup$
    – D.W. ♦
    Apr 27, 2017 at 17:48
  • 1
    $\begingroup$ @D.W. The problem is that when you use SGD for optimization, there is not an effective way to apply constraints. In addition, even if you have a way to force $\lambda\ge 0$, you get the minima at $\lambda=0$. That is why $\lambda$ can't be optimized with SGD. $\endgroup$
    – user172818
    Apr 28, 2017 at 1:24
3
$\begingroup$

You say you want to understand how $\lambda$ and $\eta$ affect the cost function.

If you hold the weights $w$ fixed, the equation for $C$ tells you how $\lambda$ affects the cost function, and $\eta$ doesn't affect the cost function; it only affects the sequence of steps taken by gradient descent.

But the tricky thing is that the final weights $w$ depend on $\lambda,\eta$. If we choose $\lambda,\eta$ and then use gradient descent to train a set of weights, the final weights $w$ will depend on $\lambda$ and $\eta$.

So, we can think of the final weights $w$ as actually being a function of $\lambda,eta$. Similarly, the final cost $C$ is actually a function of $w,\lambda,eta$, and since $w$ in turn depends on $\lambda,\eta$, this means that the final cost $C$ is really a function of the two variables $\lambda,\eta$. Suppose we write it as a function, to make the dependence clearer: $C(\lambda,\eta)$.

Now we want minimize $C(\lambda,\eta)$. If you want to minimize this using gradient descent, you need to be able to compute the partial derivatives ${\partial \over \partial \lambda} C(\lambda,\eta)$ and ${\partial \over \partial \eta} C(\lambda,\eta)$. Unfortunately, it is not at all clear how to write down an analytical expression for those partial derivatives. The tricky bit is that it is not at all clear how the final weights $w$ depend on $\lambda,\eta$ -- we have no nice way to write down an analytical expression for that. So, we can differentiate to get

$${\partial \over \partial \lambda} C(\lambda,\eta) = {\partial \over \partial \lambda} C_0(\lambda,\eta) + {\lambda \over 2n} \sum_i 2 w_i {\partial w_i \over \partial \lambda} + {1 \over 2n} \sum_i w_i^2,$$

but how do we compute ${\partial w_i \over \partial \lambda}$? And how do we compute ${\partial \over \partial \lambda} C_0(\lambda,\eta)$? It's not clear, as we don't have an analytical expression for the final weights $w_i$ as a function of $\lambda$; the only way we have to compute the weights is to run the gradient descent algorithm to completion.

So that's the challenge with using gradient descent to optimize $\lambda,\eta$, and one reason why people often use grid search instead.

I'm not saying applying gradient descent is impossible, but one must apply other black-box methods to compute the gradients, and there are other reasons why this might fail due to multiple local minima. Usually grid search is easy enough to apply and thus that is what is used.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ This is an interesting view. However, the real minima of $C(w,\lambda)$ is independent of $\eta$, a parameter used in numerical optimization. For a simpler analogy, the integral of a function is independent of the step size used in numerical integral. $\endgroup$
    – user172818
    Apr 26, 2017 at 22:35
  • $\begingroup$ @user172818, true. Ideally -- if gradient descent is finding the true global minimum -- $\eta$ wouldn't affect the value of $C(w,\lambda,\eta)$. However, in practice, the learning rate often does affect the result of gradient descent -- partly because we often use "early stopping" where we don't continue gradient descent forever, but stop after a fixed number of epochs. This is effectively a weak form of regularization / overfitting-prevention. As a result, in practice $C$ might depend on $\eta$, though your point is well taken. Anyway, I think the dependence on $\lambda$ is sufficient. $\endgroup$
    – D.W. ♦
    Apr 26, 2017 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.