Facebook announced that it would employ a machine learning "reporting process using pattern recognition in posts previously reported for suicide" (https://newsroom.fb.com/news/2017/03/building-a-safer-community-with-new-suicide-prevention-tools/).

Could anyone explain in more detail how this pattern recognition algorithm might work in their reporting system? Is it possible to say what kind of classification algorithm are they most likely using? Something similar to a spam filtering algorithm?

Here's the section I quoted from:

"Based on feedback from experts, we are testing a streamlined reporting process using pattern recognition in posts previously reported for suicide. This artificial intelligence approach will make the option to report a post about 'suicide or self injury' more prominent for potentially concerning posts like these. We’re also testing pattern recognition to identify posts as very likely to include thoughts of suicide. Our Community Operations team will review these posts and, if appropriate, provide resources to the person who posted the content, even if someone on Facebook has not reported it yet"

  • 1
    $\begingroup$ @Evil Sentiment analysis is an actual field of study, so I think this question is potentially answerable. It doesn't have to be parsed as "Does anybody know Facebook's trade secrets?" and I think an answer along the lines of "I don't know what Facebook actually uses but my research achieves something closely related by doing X, Y and Z" could be written. $\endgroup$ Commented Aug 5, 2017 at 21:30

2 Answers 2


I would take march average to calculate several indicators from posts of the same user including but not limited to:
- emotional-tagged words from posts (using available dictionary for this purpose) - sudden changes in the writting style
- response times to messages
- relationship changes
- job changes

Also tagged messages reported by users, if it abides with policy also filtering concerns from private messages (reading them is probably illegal, but feeding machine learning scripts, maybe would be ok). For the supervised learning, the morbid idea is to feed learning algorithm the posts of people who made attempts.

To feed any data it should be preprocessed (to get rid of noise - texts that have no indicating value), possibly with included data about gender, age, education. Since it may not be properly filled in, maybe some additional scavenging and recognition of profile images (to determine age fir example) may be needed.

From reading the link it looks like scheme to allow flagging concerns, also the group of experts is reviewing the reports, so it is for sure mixture of unsupervised, supervised and experts learning system, scrapping every possible data.

To answer the main question, it depends on data acquired, resources available (for example how many review hours from experts they are willing to afford). The gathered data (and assumed certainty) shapes the used algorithms. The other problem is that there are no good enough tests to tell apart depressed or suicidal people apart based only on social media or written tests, so my bet is there are at least several algorithms formed into polling system, quite like ICD-11 or DSM-V diagnostic manuals, poll-based operationslisation of state to detect disorders. But the sentiment analysis gives roughly 20% of good results when used for opinions, for emotion detection it vastly degrades, so this approach would not be considered as is for safety reasons. The Stanford NLP unit works hard on higher level understanding of text, with sentiment analysis demo available. It works poorly for short texts, peaking at moderately long ones. It may be incorporated into longitudal working (time based consecutive tests), where afaik it could yield 25% for negative mind state of user.

Your idea of spam filtering may work in this context, where filter detects anomalies of behaviour or predefined patterns comming from tagged messages, but it must yield many false positives (due to weak patterns available).

The natural choice would be RNN, as Martin wrote, to account for dynamic temporal behaviour, if any other NN is used, such feature would be simply emulated, to match temporal behaviour of suicide plans. Estimating risk factors and including personal data (some auxiliary tests are required to asses personality traits, hopefuly these are quite accurate and work even in transparent setting, when someone is evaluated from behaviour), there should exist several time scale factors independently at least for every main personality type. The main disadvantage here is accuracy of profile data, for example five years inaccuracy in the age field filled by user matters. If someone for some reasons fills birthday date wrongly, the learning system cannot depend on valuable data what user does that day and whom meets. This may seem as tangential, but users privacy is important here, if someone feels insecure, the important data is lost, otherwise inferrence is inaccurate.

The algorithms used is not arbitrary chosen, it does not matter which one is performant or accurate for some tasks, it is data dependend choice, so it is hard to pick. Another factor is assesment, which is far from ideal. We can conclude that there are non-standard algorithms created solely for this task (perhaps including several known ones), but the heavy lifting is still done by people, as the machine learning works as screening test, flagging for review.

The fresh and interesting article about Predicting Risk of Suicide Attempts Over Time Through Machine Learning claims 80-90% accuracy, but is not applicable here, because it uses medical records to work.


While I can't give a definite answer (I don't work for Facebook), I have a couple of ideas what to look for.

First of all, the area is called (Natural Language Processing) and while you could phrase it as a binary classification problem, I rather think they will apply sentiment-analysis. This might also be relevant to advertisement (you rather want ads next to positive posts)

The question What is the Best and easiest way to create a Classifer for Sentiment Analysis was closed as to broad, so the same might apply here.

I think the best systems today use RNNs (recurrent neural networks) and learn pretty much from the raw text (see How does a recurrent connection in a neural network work?).

Simpler algorithms use stemming, tf-idf features and apply a classifier to it (could be a neural network, a random forest or a SVM).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.