Tokenization

1. Tokenization In OpenAI API : Let’s Explore Tiktoken Library, Github - Pair encoding (BPE) for tokenization, BPE is a data compression algorithm.

2. (really good) Hugging Face on tokenization

3. An Explanatory Guide to BERT Tokenizer