In case of binary classification problem, what are the $y_i$ 's in the training data set $\{(x_1, y_1), (x_2, y_2), \dots (x_n, y_n)\}$?
I guess they are from $\{1,-1\}$. Now I see a method for finding a scoring function $f(x) = w^Tx + b$ by minimizing the squared error between the $f(x_i)$'s and $y_i$'s over $w$ and $b$. Now is it correct to minimize the error between $f(x_i)$'s and $y_i$? The latter is a sign while the former is a value? They seem incomparable to me.
If your prediction is $f(x) = \text{sign}(w^T x + b)$ then the 0-1 loss will simply be $$ L_{\text{0-1}}(w) = \sum_{i=1}^n \mathbb{I}\{y_i \neq f(x_i)\}, $$ where $\mathbb{I}$ is the indicator function, i.e., $\mathbb{I}\{a\} = 1$ if $a$ is true and $0$ otherwise. If you were to minimize this directly using subgradient descent you would get the perceptron algorithm. A more common approach is to upperbound the 0-1 loss with some other function, like the hinge loss, log loss, squared loss, etc and minimize that.
Actually your prediction will be sign(f(x)), not just f(x). So the prediction of the model will be +1 or -1.
