> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/natural-language-processing-nlp/tokenization.md). # Tokenization Before a language model can read a sentence, the text has to be cut into tokens, and the way it is cut decides what the model sees. The notes below start with byte pair encoding (BPE) and tiktoken, the tokenizer behind the OpenAI API, then move to the Hugging Face summary of tokenization algorithms, and end with a guide to the BERT tokenizer. The same notes are in [BERT](/natural-language-processing-nlp/pretrained-language-models.md#bert) and [GPT](/generative-ai/large-language-models-llms.md). The OpenAI side comes first. [Tokenization In OpenAI API : Let’s Explore Tiktoken Library](https://medium.com/@basics.machinelearning/tokenization-in-openai-api-lets-explore-tiktoken-library-d02d3ce94b0a) explores Tiktoken, the open-source tool OpenAI built for tokenizing text. The library itself is on [Github](https://github.com/openai/tiktoken). Under it sits pair encoding: BPE for tokenization, where BPE began as a data compression algorithm. BPE is only one of the algorithms in use, so the next stop is the wider survey. [Hugging Face on tokenization](https://huggingface.co/docs/transformers/tokenizer_summary) is the Hugging Face summary of tokenization algorithms, and it is really good. The last step narrows back to a single model. [An Explanatory Guide to BERT Tokenizer](https://www.analyticsvidhya.com/blog/2021/09/an-explanatory-guide-to-bert-tokenizer/) is Pritesh's walk through the BERT tokenizer, written from the point that the input matters for transformers and the tokenizer libraries are crucial to getting it right. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://www.mlcompendium.com/natural-language-processing-nlp/tokenization.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.