> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/data/engineering-intro/multi-cpu-processing.md).
# Multi CPU Processing
One process is not enough once NumPy and Pandas jobs get large, so this page is how those same libraries use more than one CPU. It moves from NumPy and Pandas on more than one process, through Dask, then async. The same notes are in [Pandas](/data/engineering-intro/data-science-tools.md#pandas).
The first step is to keep the libraries you already use and give them more processes. [Numpy](https://gitlab.com/tenzing/shared-array) on multi process, and [how to use it.](https://medium.com/analytics-vidhya/multiprocessing-for-data-scientists-in-python-427b2ff93af1) are the SharedArray project on GitLab and Sebastian Theiler's multiprocessing for data scientists in Python. For dataframes, [Pandas on multi process](https://github.com/nalepae/pandarallel) is pandarallel, a simple and efficient tool to parallelize Pandas operations on all available CPUs.
When one machine's processes are still not enough, Dask is the library that schedules the work. [Dask](https://docs.dask.org/en/latest/) is the Dask documentation, and the Dask youtube channel has the [intros](https://www.youtube.com/channel/UCj9eavqmvwaCyKhIlu2GaoA). Diagnostic [dashboards](https://www.youtube.com/watch?v=N_GqzcuGLCY) is the Dask Dashboard walkthrough by Matthew Rocklin, and [Distributed sklearn](https://www.youtube.com/watch?v=5Zf6DQaf7jk) is distributed scikit learn by Tom Augspurger.
Dask is not the only way to speed up an apply, and the comparison of Dask vs swifter vs vectorize (its source is kept at the end of the page) comes down to three notes:
1. Dask is dask
2. Swifter will attempt to understand if dask or pandas apply should be used, looks like its using multi cpu so it may not be just using dask on the backend?
3. Vectorize is just another option
Outside dataframes, the same idea applies to any Python job. [Multi process cpu example](https://datascience.blog.wzb.eu/2017/06/19/speeding-up-nltk-with-parallel-processing/) is the WZB Data Science Blog on speeding up NLTK with parallel processing. [Medium on MP, using MP pool, Ray etc.](https://medium.com/distributed-computing-with-ray/how-to-scale-python-multiprocessing-to-a-cluster-with-one-line-of-code-d19f242f60ff) is Edward Oakes showing how `multiprocessing.Pool` can be scaled from a single machine to a cluster.
Processes are one answer; the last one is not to wait at all. [Async (multi process/thread/coroutines/asyncio)](https://medium.com/velotio-perspectives/an-introduction-to-asynchronous-programming-in-python-af0189a88bbb) introduces asynchronous programming, where a unit of work runs separately from the main thread and notifies it when it completes or fails.
## Deprecated links
{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}
* Dask vs swifter vs vectorize. This address no longer opens:
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://www.mlcompendium.com/data/engineering-intro/multi-cpu-processing.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.