So the problem I'm solving is this: I have a list of conversations of 3 messages each (for eg. "hi", " how are you", "remind me to fix this bug" is one conversation, and my problem will have many of these) and need to pick out the message that is most likely to be a task ("remind me to fix this bug" would be the message in the conversation above; "don't forget to commit your code" would be another in some other conversation). What kind of machine learning/NLP model could I use to relatively rank the messages in a conversation and "learn" across conversations? And what would my training data look like?

My thoughts: each data point in the training data is one conversation (a set of 3 messages) along with the message with the highest likelihood of being a "task." I was thinking of using a linear weighted function of features such as TF-IDF score. But how would I learn the weights?


Build a boolean classifier that classifies a message as either "task" or "not a task". Choose a classifier that can output a confidence score (many classifiers can, including ensembles like random forests; neural networks and deep learning; logistic regression; and more).

Train the classifier on many messages.

For a collection of 3 messages, apply the classifier to each message and get the confidence score. Then, use the confidence scores to rank the 3 messages: e.g., choose the message that has the highest confidence/probability of being "task" as the one that is most likely to be the task.

This is different from what you had in mind: here each instance is a message, not a conversation.

Which features to use is a separate question, but you'll probably want to use standard NLP features (bag of words, word2vec, and so on). That belongs in a separate question. As a warning, if each message is very short as indicated in your example, the performance of your classifier might not be great.

Also, you will need a large collection of messages where you know the label (classification) of each.

  • $\begingroup$ Thank you for your answer! I have a few questions though: 1. Would you happen to know an online/C# library that could do that? 2. For classifiers such as logistic regression and neural nets, is the probability equivalent to the confidence score? (I've studied these topics before but haven't implemented a confidence score I'm sorry if this is silly, haha but I'm new!) 3. Why would short messages be bad? Would using actual message logs be a better idea? (in a way that doesn't compromise privacy) $\endgroup$ – Mathguy Jun 3 '16 at 4:19
  • 1
    $\begingroup$ @Mathguy, thanks for your interest! I'm afraid this site's format isn't so good for interactive discussions, follow-on questions, etc. I'd suggest spending some time reading about the subject and then if you still have questions, ask a new question. 1. Product recommendations are off-topic here, but there are lots of resources online. 2. Yes. 3. There's less for the ML to work with so accuracy will be lower. I don't know what message logs are so I can't help with that. $\endgroup$ – D.W. Jun 3 '16 at 4:55
  • $\begingroup$ Thank you, I now know what exactly to look for! I shall look into a good source for a specific library. $\endgroup$ – Mathguy Jun 3 '16 at 6:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.