Binary Classification- Non-Differentiable Loss Function

Question

For binary classification using linear regression, we pass the output z of the linear regression through the sigmoid function so that if the linear regression takes an input x which should be classified as 1, and maps it to $z=300$, then the process is not penalized. But a more simplistic alternative is just using a $$L_{0-1}(y,t) = |y-t|$$ loss function where $$y = \begin{cases} 0 & \text{if } z < 0.5 \\ 1 & \text{if } z \geq 0.5 \end{cases} $$ . The issue pointed out is that this function has a derivative which is zero basically everywhere, but this reasoning doesn't seem "fundamental" to me. Is the only reason we use the sigmoid function over this other function for computational purposes, and so that we can apply gradient descent?

Answer 1

The "fundamental" basis behind logistic regression is that it is based on a particular statistical model. It tries to fit a statistical model, where the log-odds of the class can be expressed as a linear function of the features. In particular, $z$ represents the log-odds, and the sigmoid function converts from log-odds to probability.

As a reminder, the odds of a particular event $E$ is $\Pr[E]/(1-\Pr[E])$. The log-odds is the logarithm of this. Therefore, the log-odds associated to an event of probability $p$ is $$\log {p \over 1-p}.$$ This function is often called the logit function, denoted $\sigma^{-1}$, i.e., $$\sigma^{-1}(p) = \log {p \over 1-p}.$$ It has an inverse, $\sigma$, which allows to convert from log-odds to probability: $$\sigma(z) = {e^z \over 1 + e^z}.$$ This function $\sigma$ is often called the logistic function or (in machine learning) the sigmoid function. The log-odds $z$ are sometimes also called a logit.

So logistic regression assumes that we compute log-odds (logit) from some weighted linear function of the features; then we convert the log-odds to a probability; then if we want to classify, we compare whether the probability is above 0.5 or not. When training, there is a nice way to train using a loss function that checks to what extent the predicted probabilities are consistent with the observed data (basically, maximum likelihood inference).

Thus, logistic regression corresponds to some clean, elegant, well-defined assumptions about the statistical process that generates the data. Empirically, that model seems to be a pretty decent approach for many datasets.

You could of course change the model. This would correspond to making different statistical assumptions about how the data is generated. For the example you list, those assumptions feel more ugly/artificial/implausible to me. So the example you list does not seem like a good approach in most applications.

But it all depends on the application what assumptions are most appropriate. See also probit regression.