1
$\begingroup$

In this post I asked about why the sigmoid/softmax function was used in classification: Binary Classification- Non-Differentiable Loss Function But I have a followup question:

We're assuming that the log-odds is a linear function of the features. This seems completely random to me. Whats the motivation for this? Why is this a reasonable assumption to make?

$\endgroup$
2
  • $\begingroup$ This is just an assumption about the model, you could actually replace the sigmoid with other choices of "activation functions". One motivation of the sigmoid function is that it is actually a softmax function where one of the inputs is fixed to 0. $\endgroup$
    – Ethan
    Commented Apr 23 at 20:36
  • $\begingroup$ The reason why probabilities are modeled as exponentials comes in some part from the literature of statistical mechanics, where the Gibbs distribution is proportional to $e^{-\beta E}$ where $E$ is the energy and in some models this energy can be linear. $\endgroup$
    – Ethan
    Commented Apr 23 at 20:37

1 Answer 1

1
$\begingroup$

It's an assumption. It might be appropriate in any particular setting, or it might not.

Logistic regression is appropriate when an additive change in a feature causes a multiplicative change in the risk (odds); and when the effects of each feature are independent. For instance, if each additional grandparent who died of cardiac disease increases my risk of dying from cardiac disease by 2x, then logistic regression might be applicable. Also, this aspect means that the coefficients of a logistic regression model have a readily-explainable interpretation (the model is interpretable), which is convenient in some settings.

A natural way to measure risk is via the odds ($p/(1-p)$). Taking the log-odds converts a multiplicative change to an additive change; i.e., a 10x increase in odds corresponds to adding $\log 10$ to the log-odds.

A common important special case is where all features are binary, i.e., every feature is either 0 or 1. Then the "additive increase causes multiplicative increase" automatically holds for each feature in isolation, and if these effects are independent, then logistic regression is suitable. Partly because this situation is so common, logistic regression is often a reasonable choice.

Of course, there are other settings where logistic regression is not appropriate and other methods are more appropriate. Logistic regression is not the only possible model. It is just one among many possible models.

Suggested background reading: https://stats.stackexchange.com/q/71176/2921, https://stats.stackexchange.com/q/48072/2921, https://stats.stackexchange.com/q/20523/2921, https://stats.stackexchange.com/q/52825/2921, https://stats.stackexchange.com/q/8106/2921

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.