2
$\begingroup$

Consider the following soft margin loss function: $\max(0, 1-yf(x))$. I have the problem of needing to compute the conditional probability $p(y|x)$ corresponding to this function and am having trouble making the connection between conditional probability and this function. I have come across a few papers (e.g. http://www.unc.edu/~yfliu/papers/lum.pdf) that say that "soft classifiers explicitly calculate the class conditional probabilities", but I do not understand how as the output of a classifier is not a probability. Can someone please explain what I am missing?

$\endgroup$
  • $\begingroup$ What do you mean by the conditional probability corresponding to this function? What are you trying to achieve? Until you can articulate your goals more clearly, I doubt you'll be able to meet them. You might also want to define your notation. (what are $x$ and $y$?) $\endgroup$ – D.W. Mar 11 '14 at 23:39
  • $\begingroup$ How about providing us with the context that the textbook provides, as a starter? $\endgroup$ – D.W. Mar 12 '14 at 1:02
  • $\begingroup$ I have deleted some comments. Let's try to keep everything civil. Abusive behavior is not tolerated here. Thanks for your cooperation. $\endgroup$ – Patrick87 Mar 12 '14 at 5:41
4
$\begingroup$

It is not possible to consistently estimate $P(y \mid x)$ if you are learning with hinge loss. Strictly proper losses, also known as strictly proper scoring rules, are the subclass of loss functions that are consistent for probability estimation. There is now a comprehensive and rich theory of proper losses, the larger class containing strictly proper losses. Without getting far into the theory of these losses, I'll give some examples/non-examples and some properties that strictly proper losses have.

The following are strictly proper losses: squared loss (least squares), log loss (logistic regression), exponential loss (boosting). The following are not strictly proper nor even proper losses: absolute loss, hinge loss (SVM).

One reason proper losses are so important is that if a loss is proper, then it is possible to map the edge $y f(x)$ to a class probability in a consistent way (if we ignore regularization). This can be done by appealing to a link function corresponding to the proper loss. In the binary classification setting, a link function maps from the interval $[0, 1]$ to the "edge space" inhabited by $y f(x)$. Given an edge $y f(x)$, the inverse link can be used to get a probability. For strictly proper margin losses (any of the strictly proper losses listed above), you can get the link function from Corollary 14 of this paper: http://jmlr.org/papers/volume11/reid10a/reid10a.pdf. This paper also discusses the problem with hinge loss for class probability estimation.

Briefly, the problem with absolute loss and hinge loss are that if a learning algorithm believes $P(Y = 1 \mid X = x) > 0.5$, then both these losses are minimized by maximizing the edge $y f(x)$. In contrast, for squared loss and other strictly proper losses, the loss is minimized when the edge (after passing it through the inverse link function) corresponds to Learner's believed $P(Y = 1 \mid X = x)$.

$\endgroup$
1
$\begingroup$

The loss function you give is the hinge loss, which is what is used by SVM. See equation (1) in the paper you link and the paragraph that immediately follows it. SVM is not a soft classifier as defined in the paper you link. Furthermore, SVMs do not estimate class probabilities, they simply define a decision boundary. If you google "estimating class probabilities svm" you'll find papers that try to do what you want. In my experience these techniques work rather poorly and if you require a class probability, you're better off using a method that estimates class probabilities. This would include things like logistic regression or neural networks (using an appropriate output and loss function of course).

$\endgroup$
  • $\begingroup$ Your answer suggests that there is some confusion inherent in my question, that I should not be looking to derive a conditional probability from the above soft margin loss function. The problem described above was taken almost verbatim from "Learning with Kernels", by Scholkopf and Smola and while I agree that most of the problems are very convoluted and confusing, I am not sure that they are so completely off-base as you suggest. $\endgroup$ – user15425 Mar 11 '14 at 15:00
  • $\begingroup$ The next part of the question from the text is, "how can you fix the problem that the probabilities $p(-1|x)$ and $p(1|x)$ must sum to 1?". The last part of the question from the text is, "How does the introduction of a third class, ('don't know') change the problem? What is the problem with this approach? Hint: What is the behavior for large $|f(x)|$?" $\endgroup$ – user15425 Mar 11 '14 at 15:03
  • 2
    $\begingroup$ There is a way to retrofit class probabilities onto an SVM: it's called Platt scaling. $\endgroup$ – Suresh Mar 11 '14 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.