What are the challenges of using gradient descent on hyper parameters λ and η to find out their optimum values? - Computer Science Stack Exchange most recent 30 from cs.stackexchange.com 2019-09-18T13:53:54Z https://cs.stackexchange.com/feeds/question/74561 https://creativecommons.org/licenses/by-sa/4.0/rdf https://cs.stackexchange.com/q/74561 0 What are the challenges of using gradient descent on hyper parameters λ and η to find out their optimum values? Kartik chincholikar https://cs.stackexchange.com/users/70743 2017-04-26T13:39:17Z 2017-06-26T15:03:27Z <p>A question from chapter 3 of Michael Nielsen's [Neural Networks and Deep Learning]:</p> <blockquote> <p>It's tempting to use <strong>gradient descent</strong> to try to learn good values for hyper-parameters such as the regularization parameter $\lambda$ and the learning rate $\eta$. Can you think of an <strong>obstacle</strong> to using gradient descent to determine values of $\lambda$ and $\eta$? </p> </blockquote> <hr> <p>What are the challenges of using gradient descent on hyper parameters <strong>λ</strong> and <strong>η</strong> to find out their optimum values?</p> <p><a href="https://i.stack.imgur.com/Tn2EN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Tn2EN.png" alt="enter image description here"></a></p> <p><strong>C</strong> is the L2 regularized cost function.</p> <p><strong>n</strong> is the length of the training data</p> <p><strong>w</strong> is the weight</p> <p>The Learning rule for the weights will be:</p> <p><a href="https://i.stack.imgur.com/dkzHs.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/dkzHs.png" alt="enter image description here"></a></p> <hr> <p>I am having trouble understanding how λ and η will affect the cost function. I know that we should decrease η as we get closer to the Cost function minima. I don't have a clue how to optimize λ. I think that the values of the hyper parameters will change for every epoch so there can't be constant optimal values.</p> https://cs.stackexchange.com/questions/74561/what-are-the-challenges-of-using-gradient-descent-on-hyper-parameters-%ce%bb-and-%ce%b7-to/74577#74577 2 Answer by D.W. for What are the challenges of using gradient descent on hyper parameters λ and η to find out their optimum values? D.W. https://cs.stackexchange.com/users/755 2017-04-26T20:41:07Z 2017-04-26T20:41:07Z <p>You say you want to understand how $\lambda$ and $\eta$ affect the cost function.</p> <p>If you hold the weights $w$ fixed, the equation for $C$ tells you how $\lambda$ affects the cost function, and $\eta$ doesn't affect the cost function; it only affects the sequence of steps taken by gradient descent.</p> <p>But the tricky thing is that the final weights $w$ depend on $\lambda,\eta$. If we choose $\lambda,\eta$ and then use gradient descent to train a set of weights, the final weights $w$ will depend on $\lambda$ and $\eta$.</p> <p>So, we can think of the final weights $w$ as actually being a function of $\lambda,eta$. Similarly, the final cost $C$ is actually a function of $w,\lambda,eta$, and since $w$ in turn depends on $\lambda,\eta$, this means that the final cost $C$ is really a function of the two variables $\lambda,\eta$. Suppose we write it as a function, to make the dependence clearer: $C(\lambda,\eta)$.</p> <p>Now we want minimize $C(\lambda,\eta)$. If you want to minimize this using gradient descent, you need to be able to compute the partial derivatives ${\partial \over \partial \lambda} C(\lambda,\eta)$ and ${\partial \over \partial \eta} C(\lambda,\eta)$. Unfortunately, it is not at all clear how to write down an analytical expression for those partial derivatives. The tricky bit is that it is not at all clear how the final weights $w$ depend on $\lambda,\eta$ -- we have no nice way to write down an analytical expression for that. So, we can differentiate to get</p> <p>$${\partial \over \partial \lambda} C(\lambda,\eta) = {\partial \over \partial \lambda} C_0(\lambda,\eta) + {\lambda \over 2n} \sum_i 2 w_i {\partial w_i \over \partial \lambda} + {1 \over 2n} \sum_i w_i^2,$$</p> <p>but how do we compute ${\partial w_i \over \partial \lambda}$? And how do we compute ${\partial \over \partial \lambda} C_0(\lambda,\eta)$? It's not clear, as we don't have an analytical expression for the final weights $w_i$ as a function of $\lambda$; the only way we have to compute the weights is to run the gradient descent algorithm to completion.</p> <p>So that's the challenge with using gradient descent to optimize $\lambda,\eta$, and one reason why people often use grid search instead.</p> <p>I'm not saying applying gradient descent is impossible, but one must apply other black-box methods to compute the gradients, and there are other reasons why this might fail due to multiple local minima. Usually grid search is easy enough to apply and thus that is what is used.</p> https://cs.stackexchange.com/questions/74561/what-are-the-challenges-of-using-gradient-descent-on-hyper-parameters-%ce%bb-and-%ce%b7-to/74579#74579 3 Answer by user172818 for What are the challenges of using gradient descent on hyper parameters λ and η to find out their optimum values? user172818 https://cs.stackexchange.com/users/8677 2017-04-26T22:09:47Z 2017-04-26T22:16:49Z <p>The cost function $C(w,\lambda)$ is <em>not</em> a function of learning rate $\eta$. You can't compute the gradient of $\eta$.</p> <p>$\lambda$ is part of the cost function. You can indeed compute its gradient and apply SGD on $\lambda$. However, when you do that, $\lambda$ will keep decreasing because its optimal value is obviously $-\infty$.</p> <p>Note that it is possible to optimize hyperparameters like $\eta$ and $\lambda$ with another neural net, but that is a different story.</p>