The following is a motivation for penalized logistic regression in some lecture notes I'm following:

Although the cost-function for logistic regression is lower bounded by 0 we get issues if the data is linearly separable.

That part I can understand it well because the logistic regression cost-function is taken to be minus some probability so that it must be greater than 0

In this case there is no finite-weight vector w which gives us this minimum cost function and if we continue to run the optimization the weights will tend to infinity. To avoid this problem, as for standard regression problems, we can add a penality term.

Why is that if that is linearly separable there is no minimum of the cost function and that the weights go to infinity?

Please ask for any details you may need on this problem. If it is any help the cost function that is being considered is:

$L(w) = \sum_{i=1}^{N} [\ln(1+\exp(x_{n}^{\top}w) - y_nx_{n}^{\top}w]$

Also the lectures notes can be find here. The issue is on page 13.

  • 2
    $\begingroup$ Please double-check your definition of the loss function. Something seems not 100% right yet (e.g., the parentheses aren't matching). Also, the link to the lecture notes doesn't work; you might want to fix the link. Finally: I suggest you try working through a simple example to see what happens. Even working in one dimension should be enough. Also, searching on "logistic regression linearly separable" turns up some explanations. $\endgroup$ – D.W. Oct 20 '16 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.