I was recently reading an article on Pearson correlation, and OLS coefficients. I came across the following section.

Finally, these are all related to the coefficient in a one-variable linear regression. For the OLS model yi≈axi with Gaussian noise, whose MLE is the least-squares problem \begin{align}\arg\min_a \sum (y_i – ax_i)^2\end{align} a few lines of calculus shows a is

\begin{align} OLSCoef(x,y) &= \frac{ \sum x_i y_i }{ \sum x_i^2 } = \frac{ \langle x, y \rangle}{ ||x||^2 } \end{align}

This looks like another normalized inner product. But unlike cosine similarity, we aren’t normalizing by y’s norm — instead we only use x’s norm (and use it twice): denominator of ||x|| ||y|| versus ||x||2.

Not normalizing for y is what you want for the linear regression: if y was stretched to span a larger range, you would need to increase a to match, to get your predictions spread out too.

I understand that using calculus we can arrive at an expression for finding a, the coefficient. The expression's denominator turns out to not contain y's norm. In the last paragraph of the excerpt, I could not understand the following line

Not normalizing for y is what you want for the linear regression

Why don't we want to normalize for y? What is the physical/geometrical significance of this?

  • $\begingroup$ Thanks! I will use LaTeX. Also, I do have provided a link to the source. $\endgroup$ – Sunit Gautam May 31 '19 at 8:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.