1
$\begingroup$

I'm trying to train some linear logistic regression models and I need regularization. My models contain around 4000 features.

I know that without regularization, a good rule of thumb is to have 10x the number of data points as the number of features.

Unfortunately I have on the order of 1000 examples. With high regularization constants, I'm still able to achieve good results. I know that I could use cross validation for choosing these constants, but I will be training these models within the inner loop of a function, and I care less about accuracy than I do about just training the model once.

Right now, I've been trying constants on the order of num_features/num_instances, and this seems to work ok. But I have no reason to believe this is good. Is there some more principled way to determine regularization constants that is not cross-validation?

I'm using scikit learn, so in practice the constant is the inverse of what it is in the literature (for example, I'm using num_instances/num_features).

$\endgroup$

1 Answer 1

1
$\begingroup$

Cross-validation is simple and easy to implement. Rather than inventing something new, which might be dubious, I recommend you use cross-validation. It works.

There are no formulas for the regularization constant, because it depends on the specific task you are trying to solve (i.e., it depends on the data).

$\endgroup$
3
  • $\begingroup$ I'm not choosing to not use cross validation because it's hard to implement. It's super simple. I know it works. I get it. I'm choosing not to use it because it has application specific drawbacks. $\endgroup$
    – Trevor
    Jun 5, 2022 at 6:42
  • $\begingroup$ @Trevor, without knowing what those application specific drawbacks are, I'm not sure whether we can give you a useful answer. Saying "give me a solution, but not cross-validation" does not make for a useful question, in my opinion. For instance, any other solution might violate the same (unspecified) requirements that cross-validation violates. I am also concerned about an XY problem if we don't know what the real requirements/constraints/goals are. $\endgroup$
    – D.W.
    Jun 5, 2022 at 20:02
  • $\begingroup$ @Trevor, I have edited my answer to explain why there are unlikely to be anything of the form you wish for. $\endgroup$
    – D.W.
    Jun 5, 2022 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.