> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/natural-language-processing-nlp/nlp.md).
# NLP Tools
The foundations only matter once a library runs them, so the practical question is which tool to pick and what data to feed it. The page starts with spaCy, then compares it with the other NLP libraries, and ends with dataset collections and embedding repositories.
#### SPACY
spaCy is the default starting point, so the first links are how to use it, how to learn it properly, and how to make it fast.
The same notes are in [Named Entity Recognition (NER)](/natural-language-processing-nlp/named-entity-recognition-ner.md) and [Swiss army knife libraries](/natural-language-processing-nlp/foundation-nlp.md#swiss-army-knife-libraries).
[Vidhaya on spacy vs ner](https://www.analyticsvidhya.com/blog/2017/04/natural-language-processing-made-easy-using-spacy-%E2%80%8Bin-python/) — tutorial + code on how to use spacy for pos, dep, ner, compared to nltk/corenlp (sner etc). The results reflect a global score not specific to LOC for example. For a structured path, the [spaCy course](https://course.spacy.io/) is "Advanced NLP with spaCy", a free online course. Once the pipeline works, SPACY OPTIMIZATION — [LP using CYTHON and SPACY.](https://medium.com/huggingface/100-times-faster-natural-language-processing-in-python-ee32033bdced) is Thomas Wolf's "100 Times Faster Natural Language Processing in Python".
### NLP Libraries
spaCy is not the only choice, and the comparisons below are about accuracy, speed, and fit for production. A page that has all the known libraries used to sit here; its address now points to an unrelated site and is kept at the end of the page.
For deep learning, "Deep Learning for NLP: SpaCy vs PyTorch vs AllenNLP" on Lucky's Notes is the [Comparison between spacy, pytorch, allenlp](https://luckytoilet.wordpress.com/2018/12/29/deep-learning-for-nlp-spacy-vs-pytorch-vs-allennlp/). spaCy's own Facts & Figures page, the hard numbers for spaCy and how it compares to other tools, is the [Comparison spacy,nltk](https://spacy.io/usage/facts-figures). For production, [Comparing Production grade nlp libs](https://www.oreilly.com/ideas/comparing-production-grade-nlp-libraries-accuracy-performance-and-scalability) is Saif Addin Ellafi's comparison of the accuracy and performance of Spark-NLP vs. spaCy, with use case recommendations. The older framing of the choice is [nltk vs spacy](https://web.archive.org/web/20201109032639/https://blog.thedataincubator.com/2016/04/nltk-vs-spacy-natural-language-processing-in-python/): the venerable NLTK has been the standard tool for natural language processing in Python, with a wide variety of tools, algorithms, and corpuses, and spaCy arrived as a competitor aiming at powerful, streamlined processing.
### NLP DATASETS
A library needs data, and a few collections gather most of it. [The bid bad](https://datasets.quantumstat.com/) is the big NLP dataset database, and the 600 [medium](https://medium.com/towards-artificial-intelligence/600-nlp-datasets-and-glory-4b0080bf5ab) post, "600 NLP Datasets and Glory", is the newsletter update on the current state of that Big Bad NLP Database. For many languages at once, [Amazon 51 Language datasets for NLU](https://www.amazon.science/blog/amazon-releases-51-language-dataset-for-language-understanding) is the MASSIVE dataset, released with the Massively Multilingual NLU (MMNLU-22) competition and workshop to help researchers scale natural-language-understanding technology to every language on Earth.
### NLP embedding repositories
Beyond raw datasets, pretrained word vectors save training from scratch.
The same notes are in [Embedding](/deep-learning/representations.md).
[Nlpl](http://vectors.nlpl.eu/repository/) is the Nordic Language Processing Laboratory word embeddings repository.
## Deprecated links
{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}
* nltk vs spacy This address no longer opens:
* Has all the known libraries. This address now points to an unrelated site:
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://www.mlcompendium.com/natural-language-processing-nlp/nlp.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.