1
$\begingroup$

I'm trying to understand how the $word2vec$ method actually nudges word vectors of similar semantic/syntactic content closer together in the word vector space.

I've read here (Quora answer) that it's because $word2vec$ tries to maximise the probability of target word $w$ given context $c$, and that this probability is roughly inversely proportional to the distance between $w$ and $c$ in the word vector space. I don't really see why the latter is the case though.

$\endgroup$
2
$\begingroup$

If two words tend to appear in the same context, then they will tend to receive a similar word vector.

For instance, a large corpus might contain sentences like "I really love Fuji apples" and "I really love Gala apples". From this we can see that "Fuji" and "Gala" have appeared with the same context (in this case the context is "love _ apples", if by context we look one word to the left and one word to the right). This will tend to cause the optimization algorithm to assign Fuji and Gala vectors that are similar.

As you can see, words that tend to be associated with the same surrounding context will often have a similar semantic content. Not always, but it's a good heuristic.

$\endgroup$
  • $\begingroup$ After accepting I realised I'm still a bit confused: your explanation shows "Fuji" and "Gala" should have similar vectors, but not necessarily that "I", "really", or "love" should be closer to "Fuji" and "Gala", than to other words. I'm guessing it's because I don't exactly see how to connect $P(w|c)$ to the inner product between $w$ and $c$. $\endgroup$ – Timsey Sep 6 '17 at 15:18
  • $\begingroup$ @Timsey, that's right. From my example "I", "really", and "love" wouldn't be closer to "Fuji" or "Gala". That too is consistent with the fact that words with similar meaning tend to have similar vectors: as we just said, this example corpus won't cause the vectors of "really" and "Fuji" to be similar, and the meaning of "really" and "Fuji" is not similar, so all is well. $\endgroup$ – D.W. Sep 6 '17 at 15:22
  • $\begingroup$ Of course, but in a larger corpus we would expect "love" to be closer to "Fuji" than say "measles" (I'm guessing). It is true that words that appear in each other's contexts are generally closer together, correct? My issue is that I don't see how that happens even with a larger corpus. $\endgroup$ – Timsey Sep 6 '17 at 15:30
  • $\begingroup$ Hm, rereading the original paper, it seems that that's not actually true after all. What really happens is that maximising $P(c|w)$ (skipgram) forces the word vector $v_w$ of word $w$ embedded from input to hidden, and $v'_c$ the word vector of context $c$ embedded from hidden to output. Since these are not the same embeddings, the word and context vectors aren't necessarily close together in a single embedding. Clearly there is some relation between embeddings, but it's not obvious (I think) how that manifests itself in either vector space. $\endgroup$ – Timsey Sep 6 '17 at 16:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.