How were the GPT-2 token embeddings constructed?

The authors mention that they used Byte Pair Encoding to construct their vocabulary. But BPE is a compression algorithm that returns a list of subword tokens that would best compress the total vocabulary (and allow rare words to be encoded efficiently).

My question is: how was that list of strings turned into the vectors that they actually used for training the model? The papers they published on the original GPT and its follow-up GPT-2 don't seem to specify those details.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.