How to figure out whether two texts refer to the same object or event

Question

Let's assume there is something happen in the world - Football world cup final. And team-1 beat team-2 with the score 3:2. So there is whole bunch of articles on every website about it, each contains slightly different wording, sometimes even on different language, but they all about the same event, and contains mostly the same info. Who won, who scored, timing, etc...

So, let's assume we have a 1000 texts about 10 different situation. What is modern approaches in cs, which could

1) group texts about the same situation together (so, instead of 1000 texts - there is 10 groups based on expressed object)

2) packing 1000 texts about the same object into 1, but which contains all new information about object. (Let's say in some text there wasn't coach interview, in some others - no timing where goals was scored, but result text should contain all unique info from each text)

Answer 1

This is a document clusterization problem. A general solution is to define some sort of distance between documents and apply a standard clusterization algorithm, such as k-means or expectation maximization. There are different approaches to picking distance measure $L$:

• Build a common vocabulary and convert each document into a bag of words, i.e. a long vector consisting of zeros and ones of length $|Vocab|$. Then let $L$ be the Hamming distance between these two representations.
• Compute word2vec embeddings of each word in a document and average them (can be also weighted by TF-IDF -- see this question on SO for details). This will produce a dense $d$-dimensional vector per each document. The distance $L$ can then be defined as Euclidean, Minkowski or more sophisticated distance on vectors in $\mathbb{R}^d$.
• Compute doc2vec embedding directly. Once again it's a dense $d$-dimensional vector per document, so the same choice of vector distances applies.