1
$\begingroup$

In most articles that compare word embeddings they use cosine distance to determine if words are similar. Why?

I guess that euclidean distance should work too. So, my question is: it doesn't? And why cosine distance doesn't fail?

$\endgroup$
1
  • $\begingroup$ The inner product is how you measure correlation in a vector space. $\endgroup$
    – Pseudonym
    Dec 16, 2021 at 23:45

2 Answers 2

1
$\begingroup$

In a certain sense, angles between embedding vectors are a more natural measure than their distances. Consider the case when your embeddings are generated using a classification neural network, and the last few operations of this networks are the following:

  • Apply the linear transformation to the embeddings: $v \to Av$
  • Predict the class by selecting the maximum coordinate: $arg\max_i (Av)_i$

In other words, you have vectors $a_1, \ldots, a_n$, where $n$ is the number of classes, and you select the max of $\langle a_1, v \rangle, \ldots, \langle a_n, v \rangle$, where $v$ is the embedding vector. Comparing two different products: $$\langle a_i, v \rangle > \langle a_j, v \rangle \iff \|a_i\| \cos(a_i,v) > \|a_j\| \cos(a_j,v)$$ Since $\|a_i\|$ and $\|a_j\|$ are fixed, the only things that actually depend on $v$ are $\cos(a_i,v)$ and $\cos(a_j,v)$. While it doesn't directly explain why $\cos(v_i,v_j)$ is a good similarity measure, it at least makes it look more natural than distances.

$\endgroup$
0
$\begingroup$

Those are two different measures of similarity. You could in principle use either one.

Both metrics are very closely connected. Let $v,w$ be two unit-length vectors (i.e., $\|v\|=\|w\|=1$), and let $d(v,w)=1-v \cdot w$ be the cosine distance between them. Then the Euclidean distance $\|v-w\|$ satisfies $$\|v-w\|^2 = 2 d(v,w).$$ In other words, if your embedding vectors are unit-length, there's effectively no meaningful difference between the two distance measures.

$\endgroup$
1
  • $\begingroup$ "if your embedding vectors are unit-length" While I can't say that it holds in general, I checked it on certain BERT embeddings some time ago, and it was indeed the case that vector lengths are similar (not unit-length though, as I recall) $\endgroup$
    – Dmitry
    Dec 17, 2021 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.