2
$\begingroup$

How do top news portals detect similar news? For example https://www.bbc.com/news/world-asia-china-51431087, if you go to this webpage, you can see the "More on this story" section at the bottom of the page. This news was published today, but it has detected similar news from days ago.

Is there any known algorithm or any library?

$\endgroup$
2
$\begingroup$

This is a proprietary system so we have no way of knowing.

In general, you will find a lot of work and potential solutions by searching for text similarity or document similarity. For starters, you could have a look at the work of Islam and Inkpen [1] on short sentence similarity (it mentions plenty of other references you can widen your search to) or a more general survey of Gomaa and Fahmy [2].

If you are interested in libraries, NLTK might be helpful.


[1] Islam, Aminul, and Diana Inkpen. "Semantic text similarity using corpus-based word similarity and string similarity." ACM Transactions on Knowledge Discovery from Data (TKDD) 2.2 (2008): 1-25.

[2] Gomaa, Wael H., and Aly A. Fahmy. "A survey of text similarity approaches." International Journal of Computer Applications 68.13 (2013): 13-18.

$\endgroup$
  • $\begingroup$ This question is like when someone asks for reference/tutorial/paper for recommendation system like Amazon or Youtube. And you cannot say that these are proprietary system and cannot say. $\endgroup$ – Nabin Feb 17 at 4:11
  • $\begingroup$ @Nabin Of course a company will not give away their trade secrets. But you can find out in general how say a recommendation system works. $\endgroup$ – Juho Feb 17 at 5:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.