> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/natural-language-processing-nlp/intro.md). # Introduction Language starts as strings, and the transformer mechanism is already behind the reader. The chapter therefore walks up from raw text: first the foundations that turn strings into something countable, then the tasks people actually build, and last the neural models that replace hand-built features with pretrained ones. After this chapter the reader can go from matching and TF-IDF to a pretrained text model and a decoding step. ## Foundations Before any model, it helps to know how little metric progress many NLP algorithms show once datasets are normalized, and [A Reality Check](/natural-language-processing-nlp/a-reality-check.md) is that warning. With expectations set, [Foundation NLP](/natural-language-processing-nlp/foundation-nlp.md) gathers the basic pipelines: chunking, collocations, stemming, phrase models, and Hebrew tools. The libraries that run those pipelines, spaCy compared with the others, plus datasets and embedding repositories, are in [NLP Tools](/natural-language-processing-nlp/nlp.md). The first concrete problem on strings is matching them. [Name Matching](/natural-language-processing-nlp/name-matching.md) handles fuzzy person-name matching with articles, public name datasets, and libraries, and [String Matching](/natural-language-processing-nlp/string-matching.md) widens that to fuzzy string matching in general and regex tooling. Once text has to be weighted rather than matched, [TF-IDF](/natural-language-processing-nlp/tf-idf.md) defines term frequency and inverse document frequency and the sparse-text trick of weighting word vectors with IDF. Those weights feed [Topics Modeling](/natural-language-processing-nlp/topics-modeling.md), the reading list that walks LSA, NMF, LDA, Mallet, coherence, lda2vec, and Top2Vec. The last foundation is how text becomes model input at all: [Tokenization](/natural-language-processing-nlp/tokenization.md) covers BPE and tiktoken, Hugging Face tokenizers, and a BERT tokenizer guide. ## Tasks With text represented, the chapter turns to the jobs built on it. [Named Entity Recognition (NER)](/natural-language-processing-nlp/named-entity-recognition-ner.md) collects milestone NER papers plus spaCy, SNER, and BiLSTM-CRF style tutorials. [Sentiment Analysis](/natural-language-processing-nlp/sentiment-analysis.md) lists sentiment databases, ground-truth rating and annotator-agreement practice, and analyzers such as VADER and TextBlob. [Question Answering](/natural-language-processing-nlp/question-answering.md) covers building Q\&A systems, image Q\&A, and retrieval backed by Milvus, and [Summarization](/natural-language-processing-nlp/summarization.md) covers extractive and abstractive methods, including TextRank-style approaches. Some tasks are about the language itself: [Language Detection Identification Generation (NLD, NLI, NLG)](/natural-language-processing-nlp/language-detection-identification-generation-nld-nli-nlg.md) covers language ID tools, neural language models, generation, translation, BLEU, and transliteration. Retrieval gets its own page in [SEARCH](/natural-language-processing-nlp/search.md), which calls out BERT-based search patterns and the recurring semantic-search problems with their fixes. [Intent Recognition](/natural-language-processing-nlp/intent-recognition.md) closes the tasks with what intent classification is and how it shows up in chatbot-style pipelines. ## Neural models The tasks above can be solved with counts and rules, but the neural path is where the chapter ends. [Neural NLP](/natural-language-processing-nlp/neural-nlp.md) groups CNN-for-text material and sequence-to-sequence resources. [Language Embeddings](/natural-language-processing-nlp/language-embeddings.md) is the reading list from foundation word vectors through sentence and document models, and [Pretrained Language Models](/natural-language-processing-nlp/pretrained-language-models.md) moves to ELMo, ULMFiT, BERT, XLNet, GPT-2, and GPT-3 with their transfer-learning and fine-tuning guides. A generative model still needs a way to pick its output, and [Decoding Algorithms For NLP](/natural-language-processing-nlp/decoding-algorithms-for-nlp.md) walks through greedy decoding, beam search, and sampling. The remaining pages stretch those models. [Multi Language](/natural-language-processing-nlp/multi-language.md) points at multilingual sentence embeddings such as LASER and cross-lingual language-model pretraining. [Augmentation](/natural-language-processing-nlp/augmentation.md) lists ways to grow text data: synonym swaps, embedding neighbors, back translation, and generation. [Knowledge Graphs](/natural-language-processing-nlp/knowledge-graphs.md) is about building knowledge graphs from text and structured sources, including automatic creation from text with spaCy. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://www.mlcompendium.com/natural-language-processing-nlp/intro.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.