5
$\begingroup$

I would like to determine interests or hobbies that people have given their Twitter timeline data. Their timeline is their historical collection of tweets.

The result I am trying to achieve is to automatically label a user with a collection of interests such as:

  • Finance
  • Photography
  • Android Development

I have tried using a topic model, LDA, using an individuals tweets as the corpus but I am not so sure about the applicability of the results. LDA gives a collection of terms associated with the topics it found. This requires some manual interpretation to distinguish what the interest is.

For example you might get some results from LDA like the following:

  • [Twitter, Java, time, work, day]
  • [code, IPhone, mobile, look, job]
  • [football, fantasy, team, pick, score]

I can generally pick out the topics people are interested in here but its just the key word or topic identifier I am interested in not the other contextual terms like team, pick or score in the last example. Ideally I would like to transform these term-based results into something like the following interests:

  • [Twitter, Java]
  • [IPhone, Mobile Development]
  • [Fantasy Football]

I have considered how to do this using a supervised approach. Given your categories of interest and using a bag-of-words model with a naive Bayes classifier the task would be to determine which category the topic terms belong to.

This requires labelled data for each category of interest which could be quite time consuming to create. Is there a possible unsupervised learning approach that could work?

I may have gone down the wrong path using topic models, if so, what existing textual research areas could help.

Not looking for a complete solution to the problem, just any suggestions, references or links would be appreciated.

Improve this question
$\endgroup$
7
  • $\begingroup$ Your question seems to be, essentially, "How can I extract semantic data from Twitter?" That's far too broad for this site: it's the sort of material that would form the basis for a book or a multi-year research project. $\endgroup$ – David Richerby Nov 22 '13 at 17:55
  • $\begingroup$ I'm voting to leave open. He is asking for any suggestions, etc. not everything. $\endgroup$ – Guy Coder Nov 22 '13 at 19:16
  • $\begingroup$ Yes I probably pitched this question too broadly and I could have left some things out like the ambiguity clause which is a different question. I will remove that. I am not looking for a complete answer that tells how to do this just a pointer to where to start. Would sentiment analysis help? Should I use an existing topic model and adjust the results or write something from scratch? I will edit the question to reflect this. $\endgroup$ – dsimmie Nov 23 '13 at 13:19
  • $\begingroup$ what's LDA? Latent Dirichlet allocation? $\endgroup$ – avi Nov 23 '13 at 14:27
  • $\begingroup$ @avi yes, have edited the title to include the full name rather than the abbreviation. $\endgroup$ – dsimmie Nov 23 '13 at 14:45
1
$\begingroup$

Thanks for your comments, reformulating the question lead me to a different way of thinking about the problem.

Getting an interest category list is actually reasonably straightforward for broad level hobbies it is just the labelled training data that may be hard to produce.

I found this paper on unsupervised text classification and this is probably a better fit than topic models in this scenario. It uses a small number of keywords (average 3) per category and segmented sentences to produce training data for a naive Bayes classifier.

I think I will use this with some Twitter specific text pre-processing and some sentiment classification to see if the categories are spoken of in a positive light.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.