# How does Gradient Descent treat multiple features?

As far as I know, when you reach the step, in a gradient descent algorithm, to calculate step_size, you calculate learning_rate * slope

Now, slope is obtained by calculating the derivative of the cost_function with respect to the feature you want to find the optimal coefficient for.

Let's say that the cost function for the purposes of this question is the sum of squared residuals.

My question is, how are coefficients of other features treated in the differentiation of the equation? For instance, if I have the equation $$y = b_0 + x_1 + x_2$$, then by calculating the derivative of the cost function with respect to $$x_1$$, one gets:

$$\frac{d}{d\left(x_1\right)}\left(\left(\hat{y}\:-\:\left(b_0\:+\:x_1+x_2\right)\right)^2\right) =$$

$$2\times\:\left(\hat{y}\:-\:b_{0\:}-x_1-x_2\right)\left(\frac{d}{d\left(x_1\right)}(x_2)\:+\:1\right)$$

In this case, how is a value obtained by substituting a value for $$x_1$$ while $$\frac{d}{d(x_1)}(x_2)$$ is still in the formula?

I watched a YouTube video (it starts at the right point) that says that $$x_2$$ is a constant (while it's a different feature) and, therefore, when differentiating, $$\frac{d}{d(x_1)}(x_2)$$ is omitted and we are left with $$2\times \:\left(\hat{y}\:-\:b_{0\:}-x_1-x_2\right)(1)$$. Is this the case or am I missing something?

• Consider that we're usually minimizing a loss function. We therefore differentiate that. In your example, we'd compute the gradient of the "sum of squared residuals" instead of the linear equation itself. Jan 30 at 22:52
• I am aware of that, the equations in my answer show the differentiation of the loss function with respect to a feature. Still, both equations (the linear equation and the loss function) contain other variables (features) as well, which raises the same question for me: are other variables (features) treated like constant in the differentiation? Jan 31 at 5:59
• Hm... I don't think I understand the question. Is $b_1$ a typo and you meant $b_0$ or $x_1$? It might be clearer if you stated the cost function explicitly for the example. You say it's an L2-loss, but later you introduce $mean$ without explaining what that is, exactly. IF you use the L2 loss, the loss function will be a squared sum of values, not a single $mean$. Jan 31 at 14:57
• @Simon Apologies, I must've gotten lost while writing the question. I've edited the question and hopefully is clearer now. I also left a link to the cost function i'm referring to. Jan 31 at 15:12
• Actually, while trying to explain the question and while studying the topic a bit further I think I can answer my own question now. $b_0$ is the y-intercept, and $x_1$ and $x_2$ are two features. My question was why do we treat all but one feature (or the intercept) as constants while differentiating with respect to that particular feature but I guess the answer is because, at that instance, we do not care about other features, only the rate of change of error with respect to only one feature and you do that iteratively for all features. Jan 31 at 17:47

That's correct. The derivative of $$x_2$$ with respect to $$x_1$$ is 0.