2
$\begingroup$

In case of binary classification problem, what are the $y_i$ 's in the training data set $\{(x_1, y_1), (x_2, y_2), \dots (x_n, y_n)\}$?

I guess they are from $\{1,-1\}$. Now I see a method for finding a scoring function $f(x) = w^Tx + b$ by minimizing the squared error between the $f(x_i)$'s and $y_i$'s over $w$ and $b$. Now is it correct to minimize the error between $f(x_i)$'s and $y_i$? The latter is a sign while the former is a value? They seem incomparable to me.

$\endgroup$
2
$\begingroup$

If your prediction is $f(x) = \text{sign}(w^T x + b)$ then the 0-1 loss will simply be $$ L_{\text{0-1}}(w) = \sum_{i=1}^n \mathbb{I}\{y_i \neq f(x_i)\}, $$ where $\mathbb{I}$ is the indicator function, i.e., $\mathbb{I}\{a\} = 1$ if $a$ is true and $0$ otherwise. If you were to minimize this directly using subgradient descent you would get the perceptron algorithm. A more common approach is to upperbound the 0-1 loss with some other function, like the hinge loss, log loss, squared loss, etc and minimize that.

$\endgroup$
1
$\begingroup$

Actually your prediction will be sign(f(x)), not just f(x). So the prediction of the model will be +1 or -1.

$\endgroup$
  • $\begingroup$ But, then how to optimize expression which involves sign. I need to find $w$. $\endgroup$ – RIchard Williams Feb 2 '14 at 13:34
  • $\begingroup$ find w, by optimizing f(x). It will give you a decision boundary. Now to classify new data (let x), use sign(f(x)). $\endgroup$ – user13070 Feb 2 '14 at 14:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.