Tokenization

Before a language model can read a sentence, the text has to be cut into tokens, and the way it is cut decides what the model sees. The notes below start with byte pair encoding (BPE) and tiktoken, the tokenizer behind the OpenAI API, then move to the Hugging Face summary of tokenization algorithms, and end with a guide to the BERT tokenizer.

The same notes are in BERT and GPT.

The OpenAI side comes first. Tokenization In OpenAI API : Let’s Explore Tiktoken Library explores Tiktoken, the open-source tool OpenAI built for tokenizing text. The library itself is on Github. Under it sits pair encoding: BPE for tokenization, where BPE began as a data compression algorithm.

BPE is only one of the algorithms in use, so the next stop is the wider survey. Hugging Face on tokenization is the Hugging Face summary of tokenization algorithms, and it is really good.

The last step narrows back to a single model. An Explanatory Guide to BERT Tokenizer is Pritesh's walk through the BERT tokenizer, written from the point that the input matters for transformers and the tokenizer libraries are crucial to getting it right.