Why word embeddings are compared with cosine distance and not euclidean?

Question

In most articles that compare word embeddings they use cosine distance to determine if words are similar. Why?

I guess that euclidean distance should work too. So, my question is: it doesn't? And why cosine distance doesn't fail?

Answer 1

In a certain sense, angles between embedding vectors are a more natural measure than their distances. Consider the case when your embeddings are generated using a classification neural network, and the last few operations of this networks are the following:

• Apply the linear transformation to the embeddings: $v \to Av$
• Predict the class by selecting the maximum coordinate: $arg\max_i (Av)_i$

In other words, you have vectors $a_1, \ldots, a_n$, where $n$ is the number of classes, and you select the max of $\langle a_1, v \rangle, \ldots, \langle a_n, v \rangle$, where $v$ is the embedding vector. Comparing two different products: $$\langle a_i, v \rangle > \langle a_j, v \rangle \iff \|a_i\| \cos(a_i,v) > \|a_j\| \cos(a_j,v)$$ Since $\|a_i\|$ and $\|a_j\|$ are fixed, the only things that actually depend on $v$ are $\cos(a_i,v)$ and $\cos(a_j,v)$. While it doesn't directly explain why $\cos(v_i,v_j)$ is a good similarity measure, it at least makes it look more natural than distances.

Answer 2

Those are two different measures of similarity. You could in principle use either one.

Both metrics are very closely connected. Let $v,w$ be two unit-length vectors (i.e., $\|v\|=\|w\|=1$), and let $d(v,w)=1-v \cdot w$ be the cosine distance between them. Then the Euclidean distance $\|v-w\|$ satisfies $$\|v-w\|^2 = 2 d(v,w).$$ In other words, if your embedding vectors are unit-length, there's effectively no meaningful difference between the two distance measures.