2
$\begingroup$

How were the GPT-2 token embeddings constructed?

The authors mention that they used Byte Pair Encoding to construct their vocabulary. But BPE is a compression algorithm that returns a list of subword tokens that would best compress the total vocabulary (and allow rare words to be encoded efficiently).

My question is: how was that list of strings turned into the vectors that they actually used for training the model? The papers they published on the original GPT and its follow-up GPT-2 don't seem to specify those details.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.