I'm trying to train some linear logistic regression models and I need regularization. My models contain around 4000 features.
I know that without regularization, a good rule of thumb is to have 10x the number of data points as the number of features.
Unfortunately I have on the order of 1000 examples. With high regularization constants, I'm still able to achieve good results. I know that I could use cross validation for choosing these constants, but I will be training these models within the inner loop of a function, and I care less about accuracy than I do about just training the model once.
Right now, I've been trying constants on the order of num_features/num_instances, and this seems to work ok. But I have no reason to believe this is good. Is there some more principled way to determine regularization constants that is not cross-validation?
I'm using scikit learn, so in practice the constant is the inverse of what it is in the literature (for example, I'm using num_instances/num_features).