1
$\begingroup$

I am creating a software that computes the cosine similarity between 2 strings.

I am using Spark as I have to use Java. I created a Bag of words model and computed the cosine similarity successfully.

Feeding the same sentences to the software and substituting the bag of words model with a TF-IDF the similarity between sentences took a hit. Some similar sentences became not similar.

Example

"a b c"

"a a b c d"

Cosine similarity using Bag of words is 0.871

Cosine similarity using TF-IDF is 0

Is this behavior correct or am I doing something wrong?

$\endgroup$
1
$\begingroup$

From the documentation:

Since logarithm is used, if a term appears in all documents, its IDF value becomes 0.

So if you're only considering similarity between two strings, then you only have one document in your corpus, and either your TF value will be 0 (since the word doesn't appear in the document), or the IDF value will be 0 (since the word does appear in the document), thus all terms in TF-IDF will be 0.

If you just have two strings, then cosine similarity should be enough for you - IDF is attempting to weigh the relative importance of each word within the entire corpus (from an information theoretic perspective, it basically treats the word as appearing uniformly across the documents it appears in within the corpus - under that assumption, one can consider IDF to be a form of information gain; i.e., how many bits of information does having this term gain you, relative to selecting a document at random from within the whole corpus. If every document in the corpus has a term, then under that uniform model, providing that term in the query doesn't help you at all in narrowing down your selection).

$\endgroup$
  • $\begingroup$ So it is expected to get 0 values because the terms exist in all the documents and the documents are not that many. $\endgroup$ – LonsomeHell May 29 '17 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.