1
$\begingroup$

As far as I know, when you reach the step, in a gradient descent algorithm, to calculate step_size, you calculate learning_rate * slope

Now, slope is obtained by calculating the derivative of the cost_function with respect to the feature you want to find the optimal coefficient for.

Let's say that the cost function for the purposes of this question is the sum of squared residuals.

My question is, how are coefficients of other features treated in the differentiation of the equation? For instance, if I have the equation $y = b_0 + x_1 + x_2$, then by calculating the derivative of the cost function with respect to $x_1$, one gets:

$\frac{d}{d\left(x_1\right)}\left(\left(\hat{y}\:-\:\left(b_0\:+\:x_1+x_2\right)\right)^2\right) =$

$2\times\:\left(\hat{y}\:-\:b_{0\:}-x_1-x_2\right)\left(\frac{d}{d\left(x_1\right)}(x_2)\:+\:1\right)$

In this case, how is a value obtained by substituting a value for $x_1$ while $\frac{d}{d(x_1)}(x_2)$ is still in the formula?

I watched a YouTube video (it starts at the right point) that says that $x_2$ is a constant (while it's a different feature) and, therefore, when differentiating, $\frac{d}{d(x_1)}(x_2)$ is omitted and we are left with $2\times \:\left(\hat{y}\:-\:b_{0\:}-x_1-x_2\right)(1)$. Is this the case or am I missing something?

Improve this question
$\endgroup$
6
  • $\begingroup$ Consider that we're usually minimizing a loss function. We therefore differentiate that. In your example, we'd compute the gradient of the "sum of squared residuals" instead of the linear equation itself. $\endgroup$
    – Simon
    Jan 30, 2021 at 22:52
  • $\begingroup$ I am aware of that, the equations in my answer show the differentiation of the loss function with respect to a feature. Still, both equations (the linear equation and the loss function) contain other variables (features) as well, which raises the same question for me: are other variables (features) treated like constant in the differentiation? $\endgroup$
    – alexandrosangeli
    Jan 31, 2021 at 5:59
  • 1
    $\begingroup$ Hm... I don't think I understand the question. Is $b_1$ a typo and you meant $b_0$ or $x_1$? It might be clearer if you stated the cost function explicitly for the example. You say it's an L2-loss, but later you introduce $mean$ without explaining what that is, exactly. IF you use the L2 loss, the loss function will be a squared sum of values, not a single $mean$. $\endgroup$
    – Simon
    Jan 31, 2021 at 14:57
  • 1
    $\begingroup$ @Simon Apologies, I must've gotten lost while writing the question. I've edited the question and hopefully is clearer now. I also left a link to the cost function i'm referring to. $\endgroup$
    – alexandrosangeli
    Jan 31, 2021 at 15:12
  • 1
    $\begingroup$ Actually, while trying to explain the question and while studying the topic a bit further I think I can answer my own question now. $b_0$ is the y-intercept, and $x_1$ and $x_2$ are two features. My question was why do we treat all but one feature (or the intercept) as constants while differentiating with respect to that particular feature but I guess the answer is because, at that instance, we do not care about other features, only the rate of change of error with respect to only one feature and you do that iteratively for all features. $\endgroup$
    – alexandrosangeli
    Jan 31, 2021 at 17:47

1 Answer 1

Reset to default
1
$\begingroup$

That's correct. The derivative of $x_2$ with respect to $x_1$ is 0.

A little context: with words like derivative and slope, you are describing how gradient descent works in one dimension (with only one feature / one value to optimize). In multiple dimensions (multiple features / multiple variables you are trying to optimize), we use the gradient and update all of the variables in each step. That said, yes, this is basically equivalent to separately updating each variable in the one-dimensional way that you describe.

Improve this answer
$\endgroup$
1
  • $\begingroup$ @alexandrosangeli, I don't understand what you're asking. Gradient descent is used to optimize an objective function, i.e., find values for the variables that minimize its value. See en.wikipedia.org/wiki/Gradient_descent. This site's format isn't designed well for interactive or back-and-forth conversation, so if you have a follow-up question, I suggest researching it, then asking a new question (making sure to provide enough context that people can understand what you are asking). $\endgroup$
    – D.W. ♦
    Jan 31, 2021 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.