I'm working on a kNN classifier to classify whether a text is written by a man or a woman on the basis of the most frequent words. However, a kNN doesn't show which features were the most important in the classification. I got the suggestion to fit a logistic regression on my training data and labels to see which words were the most important in the classification task.

Now I have a list of the top 20 positive words and coefficients and top 20 words with negative coefficients. I do not know how to interpret these. I can see that the logistic regression has the classes F and M (don't know which one is 0 and which is 1), but is it possible to argue that the positive coefficients belong to the F class and the negative belong to the M class?

sample output

  • $\begingroup$ Write the equation for the classifier and see for yourself. $\endgroup$ – Yuval Filmus May 29 '17 at 10:31
  • $\begingroup$ Do you remove the stop words? $\endgroup$ – Marlysson May 29 '17 at 11:51
  • $\begingroup$ What research have you done? How to compute feature importance / feature rankings for logistic regression is well documented in lots of places (including in multiple places on Cross Validated). $\endgroup$ – D.W. May 30 '17 at 1:57

I don't have the rep to comment, but the answer to your question is "sort of." It would take me too long to answer this question completely and with the proper care, so i will point you to a good resource http://www.appstate.edu/~whiteheadjc/service/logit/intro.htm

There is a section midway through on interpreting your regression coefficients. In your specific case (as I imagine all of your independent variables only take on positive values), your interpretation is fine. If $b_1$ is the slope coefficient of an indicator variable for some word, then a 1 unit change in $b_1$ (i.e. if the word corresponding to $b_1$ is seen) corresponds to an $\exp{(b_{1})}$ change in the log odds of your model (e.g. the probability of the person being a boy is $\exp{(b_{1})}$ more likely). Hope this helps

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.