> This is part 1 of 3 of the full documentation (pages 1–100 of 249).
> The content is paginated: fetch every part to see all of it.
> Next part: https://www.mlcompendium.com/llms-full.txt/1
> Page index: https://www.mlcompendium.com/llms.txt

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
This site has an agentic ask interface you may use to query the documentation dynamically by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://www.mlcompendium.com/index.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

---

# AI Compendium

The AI Compendium was previously the Machine & Deep Learning Compendium, also known as the ML Compendium and the ML & DL Compendium. It started as a private list of resources and became an open book that walks from a business problem through data, AI, engineering, and product. This page says what the book covers, where it lives, and how it came to be, and the reading order itself is in [Introduction](/readme/introduction).

The book was reorganized once already, and the record of that makeover is kept so the old order can still be traced. A makeover of this book was done. [How it was done](https://github.com/orico/www.mlcompendium.com/tree/main/appendix/makeover.md).

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2FouHuBLPAPIHcXpZQGP5K%2Fimage.png?alt=media&amp;token=31ec0b47-2d6c-49ee-914e-592eed426e5d" alt=""><figcaption><p>The Machine and Deep Learning Compendium.</p></figcaption></figure>

The scope is wide on purpose. Covering **502 topics**, the ML and DL Compendium includes summaries, links, and articles across a wide array of subjects, including LLMs. These range from modern machine learning algorithms and deep learning techniques to specialized areas like NLP, audio processing, computer vision (classic and deep), time-series analysis, anomaly detection, and graphs. It also goes deep into strategic themes like data science management, team building, and practical essentials like product management, design, and technology stacks from a data science perspective. The [Ops](/ai-engineering/intro) half of this book covers how you build, ship, govern, and run the system.

Because the book is for everyone, the source is open too. The ML and DL Compendium is completely open and now lives on [GitHub](https://github.com/orico/www.mlcompendium.com/) (please star it!). Driven by my belief in knowledge-sharing and education, this project will always remain not-for-profit and free.

{% embed url="<https://github.com/orico/www.mlcompendium.com>" %}
The ML and DL Compendium official GitHub repo
{% endembed %}

The repo is where the book is kept; the reason it exists is older. The Machine and Deep Learning Compendium began as a personal project: a curated list of resources I maintained in a private Google document for my own learning. That document has now evolved into this new interface, and I am excited to share it as an educational tool to help others learn and connect with the brilliant authors I have summarized, quoted, and referenced.

I envision it as a go-to resource for learners of all levels, whether you are an industry data scientist, an academic, or just starting out. It is designed to save you countless hours of searching and filtering through content, providing a streamlined path to invaluable authors and resources you can further support.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2FH5MeNsH1ACCNTJ0qq1iM%2Fimage.png?alt=media&amp;token=2a94f84c-1821-4fd1-b974-3194a48d389b" alt=""><figcaption><p>The Machine and Deep Learning Compendium.</p></figcaption></figure>

Let us work together to support the community, amplify the voices of authors, and democratize education. If you spot something that could be improved, feel free to contribute via [GitHub](https://github.com/orico/www.mlcompendium.com/tree/master) or [reach out](https://www.linkedin.com/in/cohenori/) to me directly.

The book also has its own article, embedded below, for readers who want the story of the compendium in one place.

{% embed url="<https://cohenori.medium.com/the-last-machine-deep-learning-compendium-youll-ever-need-dc973643c4e1>" %}
The ML Compendium article, by Dr. Ori Cohen
{% endembed %}

Many thanks,

Dr. Ori Cohen

[My Website](https://www.oricohen.com/) | [Medium](https://medium.com/@cohenori) | [LinkedIn](https://www.linkedin.com/in/cohenori/) | [ML Compendium](http://www.mlcompendium.com/) | [Ops](/ai-engineering/intro) | [State of GenAI](https://stateofgenai.com/) | [State Of MLOps](https://stateofmlops.com/) |


# Introduction

The AI Compendium was previously the Machine & Deep Learning Compendium, also known as the ML Compendium and the ML & DL Compendium. The old order is the order the notes were collected, and this order is the order of the work. An AI engineer solves a business problem with data, AI, and engineering, and puts the result in a product, so the model is one step on that path.

The first diagram shows the change: the notes used to run from types of learning to a separate ops book, and now they run from the business problem to the product.

```mermaid
flowchart TD
 subgraph collected [How the notes were collected]
 types[Types of learning]
 foundations[Foundations mixed with management]
 methods[Methods filed by folder]
 opsHalf[Ops as a second book]
 types --> foundations --> methods --> opsHalf
 end
 subgraph written [How the book is written]
 business[Business problem]
 data[Data]
 ai[AI]
 engineering[AI Engineering]
 product[AI Product]
 business --> data --> ai --> engineering --> product
 end
 collected --> written
```

The second diagram opens the Data step into its four parts, because most of the work between the business question and the model is there.

```mermaid
flowchart LR
 business[Business problem]
 datasets[Datasets]
 processing[Data Processing]
 analytics[Analytics]
 dataEng[Data Engineering]
 ai[AI]
 engineering[AI Engineering]
 product[AI Product]
 business --> datasets --> processing --> analytics --> dataEng --> ai --> engineering --> product
```

The notes inside each chapter are the sources for a later manuscript, and two posts explain where that manuscript comes from. [The ML & Ops Compendium Open Books](https://medium.com/data-science-collective/the-ml-ops-compendium-open-books-43a3c571d971) (December 2024) explains the open books this manuscript grows from. The earlier one, the August 2021 post titled The Machine & Deep Learning Compendium Open Book, is pinned on the [author profile](https://cohenori.medium.com/) and belongs in this introduction.


# Thanks Page

A compendium built from other people's work is also improved by other people. This page names the contributors who helped shape it.

I would like to thank the following contributors: [Samuel Jefroykin](https://www.linkedin.com/in/samueljefroykin/), [Sefi Keller](https://www.linkedin.com/in/sefikeller/?originalSubdomain=il)


# Introduction

This chapter is the job, before the data and the method. After it, the reader can state what the business wants changed and how that metric will sit beside a model score, and the worked cases are in [AI Product](/ai-product/intro).

The chapter has one page. [Data Science](/business-problems/data-science) walks the role of a data scientist, the project lifecycle, the platforms and stack, and how a research team plans its work, and only then the courses, books, cost, and patents.


# Data Science

A company hires a data scientist to turn a business problem into a model someone else can run. That only works if the role, the lifecycle, the stack, and the way the team works are named before anyone picks a course or a book. The path below starts with the job itself, then the process and the platforms, then how a research team actually moves, and only then the courses, books, cost, and patents.

## Being a DS / Researcher

The job is a day of engineering mixed with research, and the first mistake is treating a business KPI as if it were a research KPI. [A day in a life](https://medium.com/data-science/12-things-i-learned-during-my-first-year-as-a-machine-learning-engineer-2991573a9195) is the first-year machine-learning-engineer account of that mix. [Advice for a ds](https://medium.com/the-data-experience/building-a-data-pipeline-from-scratch-32b712cfb1db) is the pipeline built from scratch, where business KPIs and research KPIs are not the same thing. A review of deep learning papers and co-authorship used to sit here; that address no longer opens and is kept at the end of the page.

Once the day is clear, the work is a stack, not a single model. The full-stack DS diagram is by [Uri Weiss](https://linkedin.com/in/uriweiss).

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-1ad748f8f975f7e16cd6bc81544efcc2de9dac0a%2Fgimg-6d9d594fee8e.png?alt=media" alt=""><figcaption><p>Full stack DS diagram.</p></figcaption></figure>

If the credit to [Uri Weiss](https://linkedin.com/in/uriweiss) on that figure is wrong, [please contact me](mailto:ori@oricohen.com). The practices that sit under the diagram, the engineering habits a DS still has to keep, are on the [ML practices for a DS](https://se-ml.github.io/) page for software engineering for machine learning.

## Life cycle

The role only becomes a team process when someone writes down how a project moves from question to deployed model. The same notes are in [MLOps Intro](/ai-engineering/mlops-intro).

[Microsoft on Team DS Lifecycle](https://docs.microsoft.com/en-us/azure/architecture/data-science-process/overview) is that write-up. Microsoft describes the Team Data Science Process as an agile, iterative methodology for predictive analytics and intelligent applications. It says how roles work together, pulls in practices from Microsoft and other companies, and aims at an analytics program that actually ships. The page gives a generic process that can be implemented with different tools, then a more detailed account of tasks and roles, then guidance for the Microsoft tools their own teams use.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-e55db9bf6314064681c8d3ddada541de28e8c8f6%2Fgimg-7581c564f5fb.png?alt=media" alt=""><figcaption><p>The DS lifecycle, Microsoft Documentation.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Mgd48oS5_duTKOVE_Et%2F-Ml8YBZabGUKuJTw7T2_%2F-Ml8ZLcCPvUpLvf_eHJn%2Fimage.png?alt=media&amp;token=78e935c4-28a5-4f46-86e2-00d3ce43e2ec" alt=""><figcaption><p>The DS lifecycle, Microsoft Documentation.</p></figcaption></figure>

Google draws the same lifecycle and then shows why the model code is the small part. [Google’s famous MLops](https://cloud.google.com/solutions/machine-learning/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning#mlops_level_0_manual_process) is the continuous-delivery picture, starting from a fully manual process.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-93b16c14ec9a55c11090740c6b60d5c658f8d6df%2Fgimg-78f852e0ca68.png?alt=media" alt=""><figcaption><p>ML systems is more than ML code. Google.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-e7ef11015fdc22383b76e8d1b865ffa8ba2cf6e5%2Fgimg-c7f80cc044e4.png?alt=media" alt=""><figcaption><p>ML systems is more than ML code. Google.</p></figcaption></figure>

Before any of that machinery, the project still needs a questionnaire. The [Fast ai project checklist](https://www.fast.ai/posts/2020-01-07-data-questionnaire.html) is the consulting list for the organization's context:

> "When I used to do consulting, I’d always seek to understand an organization’s context for developing data projects, based on these considerations:
>
> * Strategy: What is the organization trying to do (objective) and what can it change to do it better (levers)?
> * Data: Is the organization capturing necessary data and making it available?
> * Analytics: What kinds of insights would be useful to the organization?
> * Implementation: What organizational capabilities does it have?
> * Maintenance: What systems are in place to track changes in the operational environment?
> * Constraints: What constraints need to be considered in each of the above areas?"

## Workflows

A lifecycle is a map. A workflow is one project walking it. After Microsoft and Google, the concrete move is a competition-style loop: the [kaggle](https://medium.com/data-science/my-secret-sauce-to-be-in-top-2-of-a-kaggle-competition-57cff0677d3c) write-up of what it took to land in the top 2% of a competition.

## Platforms

A workflow still has to run somewhere. The tour of [Uber, google, netflix, airbnb, etc](https://databaseline.tech/a-tour-of-end-to-end-ml-platforms/) is how those companies built the platform that holds the workflow, not just the notebook.

## Stack

The platform is made of a small set of pieces that show up again and again. [Medium on canonical stack](https://medium.com/data-science/rise-of-the-canonical-stack-in-machine-learning-724e7d2faa75) is the argument that machine learning converged on a canonical stack.

## Agile for data-science-research

A stack does not tell a research team how to plan the week. Research is not a backlog of known tickets, so the method here is agile without pretending it is scrum or kanban. The same notes are in [Building Teams](/ai-product/management#building-teams) and [Scaling Agile - Agile Approaches](/ai-product/management#scaling-agile---agile-approaches).

[How to manage a data science research team using agile methodology, not scrum and not kanban](https://medium.com/data-science/data-science-agile-cycles-my-method-for-managing-data-science-projects-in-the-hi-tech-industry-b289e8a72818) is the cycle used for high-tech research projects. [Workflow for data science research projects](https://medium.com/data-science/data-science-project-flow-for-startups-282a93d4508d) is the startup version of that flow. [Tips for data science research management](https://medium.com/data-science/my-best-tips-for-agile-data-science-research-b40365cc979d) are the operating tips that go with it. [IMO a really bad implementation of agile for data-science-projects](https://www.locallyoptimistic.com/post/agile-analytics-p1/) is Michael Kaminsky's "Agile Analytics, Part 1: The Good Stuff" on Locally Optimistic, kept here as the counterexample: what happens when analytics is forced into a ceremony that does not fit the work.

## Team Building / Group Cohesion

The method only holds if the team knows which job is which. The same notes are in [Building Teams](/ai-product/management#building-teams) and [Scaling Agile - Agile Approaches](/ai-product/management#scaling-agile---agile-approaches).

[DS vs DA vs MLE](https://medium.com/@meightpc_14421/data-scientist-vs-data-analysis-vs-ml-engineer-which-job-is-most-suited-for-you-def7b12b3256) is the diagram-heavy comparison of data scientist, data analyst, and ML engineer, a distinction that is the motherlode of the figures people reuse when they argue about the role.

The team then has to form. The references are a roadmap and the classic stages, not a new org chart:

[1](https://medium.com/@rdavila01/a-team-development-roadmap-ce5247127037) is a team-development roadmap. [2](https://medium.com/swlh/team-development-stages-51df5606c0a2) and [3](https://medium.com/unexpected-leadership/forming-storming-norming-and-performing-5d06d021a969) are the forming, storming, norming, and performing stages. [4](https://medium.com/@RiterApp/8-models-of-team-effectiveness-3a3b84efb3ae) lists eight models of team effectiveness. [5](https://medium.com/@warren2lynch/traditional-to-scrum-team-forming-storming-norming-and-performing-3fd5fd1f5ea9) carries those stages from a traditional team into scrum. [6](https://medium.com/@pallawi.ds/new-employee-best-practices-to-perform-with-the-team-tuckmans-stages-of-group-development-c656ca295bee) is Tuckman's stages from the new employee's side. [7](https://medium.com/agilegreat/tuckman-model-for-building-great-teams-7b3203d7a9e3) is the Tuckman model aimed at building the team. [8](https://medium.com/simply-agile/agile-leader-pattern-2-for-building-awesome-teams-stabilize-teams-32785b70868c) is the agile-leader pattern of stabilizing teams. [9](https://medium.com/hackernoon/team-building-mental-models-1f431ae29361) is the mental models behind team building. Item 10 in that run has no address left on the page.

The hiring implication of those stages is [Why data science needs generalists not specialists](https://hbr.org/2019/03/why-data-science-teams-need-generalists-not-specialists). [(great) Building a DS function (team)](https://medium.com/ww-tech-blog/from-0-to-60-models-in-two-years-building-out-an-impactful-data-science-function-9ef86abb9605) is the story of standing that function up, from zero to sixty models in two years.

## Culture

A formed team still has a culture that decides who stays. Netflix states the aspiration directly: entertain the world, thrilling audiences everywhere, on the [Netflix](https://jobs.netflix.com/culture) culture page. [Reed hastings on netflix' keeper test](https://hrtechx.com/2020/11/20/netflixs-keeper-test-is-the-secret-to-a-successful-workforce/) is the keeper test as the workforce mechanism behind that page. A [response 1](https://www.highlights.lornerubis.com/page/83/) to that test used to be linked here; the address no longer opens and is kept at the end.

## Building Data/DS teams

Culture then has to become a structure. The same notes are in [Data Teams](/data/engineering-intro/data-teams) and [MLOps Teams](/ai-engineering/mlops-teams).

[(great) the data team a short story by erik bern](https://erikbern.com/2021/07/07/the-data-team-a-short-story.html) is Erik Bernhardsson's story of being brought into a startup to run a three-person data team. [Guilds / Gangs / Squads](https://aviranm.medium.com/the-evolution-of-a-guild-a6c7d1927610) is Aviran Mordo on how a guild evolves. [Squads, Tribes, Guilds, dont be like Spotify](https://uxdesign.cc/squads-tribes-guild-to-be-like-spotify-or-not-13ecf690fd36) is the warning against copying the names without the conditions. [Discover the Spotify Model](https://www.atlassian.com/agile/agile-at-scale/spotify) is Atlassian's account of the model Spotify used to scale agile by putting culture and network ahead of a fixed hierarchy.

## SOTA and current trends summaries

A team that can ship still has to know what the field just changed. Chip Huyen's [ICLR 2019](https://huyenchip.com/2019/05/12/top-8-trends-from-iclr-2019.html) is a Twitter-thread-style list of eight trends, with the disclaimer that it does not reflect the organizations she is associated with and is peppered with personal and institutional biases. The [Medium](https://medium.com/huggingface/the-best-and-most-current-of-modern-natural-language-processing-5055f409a1d1) piece is Hugging Face on what was current in modern NLP at the time. [State of ai, a yearly report](https://www.stateof.ai/) is the independent annual report covering research, industry, geopolitics, and safety, the recurring summary after the one-off conference notes.

## YouTube courses

The practice above is what the courses are for. This shelf is the long-form video and tutorial path, and it is deliberately mixed: some of it is excellent, some of it is too long or not intuitive, and that judgment stays with the link.

[DEEPNET.TV YOUTUBE (excellent)](https://www.youtube.com/channel/UC9OeZkIwhzfv-_Cb7fCikLQ) is the channel to start with. [Mitchel ML Lectures (too long)](http://www.cs.cmu.edu/~ninamf/courses/601sp15/lectures.shtml) are the Carnegie Mellon 10-601 lectures. [Quoc Les (google) wrote DNN tutorials and 3H video (not intuitive)](http://cs.stanford.edu/~quocle/) is Quoc Viet Le's page, useful and not the intuitive introduction. [KDnuggets: numpy, panda, scikit, tutorials.](http://www.kdnuggets.com/2015/11/seven-steps-machine-learning-python.html) is the seven-step path through the Python stack. [Deep learning online book (too wordy)](http://neuralnetworksanddeeplearning.com/) is the neural networks and deep learning book credited to David Cervone, and it is the wordy one. [Genetic Algorithms - grid search hyper params better than brute force.. obviously](https://medium.com/@harvitronix/lets-evolve-a-neural-network-with-a-genetic-algorithm-code-included-8809bece164) is evolving a network instead of brute-force search. [CNN tutorial](http://mccormickml.com/2015/01/10/understanding-the-deeplearntoolbox-cnn-example/) walks the convolutional example in Rasmus Berg Palm's DeepLearnToolbox. [Introduction to programming in scikit](http://nbviewer.jupyter.org/github/donnemartin/data-science-ipython-notebooks/blob/master/scikit-learn/scikit-learn-intro.ipynb) is the notebook introduction. From the same PyCon 2015 materials, [SVM in scikit python](https://github.com/jakevdp/sklearn_pycon2015/blob/master/notebooks/03.1-Classification-SVMs.ipynb) is the classifier notebook and [Sklearn scipy PCA tutorial](https://github.com/jakevdp/sklearn_pycon2015/blob/master/notebooks/04.1-Dimensionality-PCA.ipynb) is the PCA notebook. [RNN](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) is colah's explanation of LSTM networks, the point where the course shelf reaches sequence models. [Matrix Multiplication](http://www.mathwarehouse.com/algebra/matrix/multiply-matrix.php) is the visual reminder of the operation under all of the above.

## Deep learning Course

The video shelf is not a course with exercises. Kadenze is. [Kadenze - deep learning tensor flow](https://www.kadenze.com/courses/creative-applications-of-deep-learning-with-tensorflow-iv/sessions/introduction-to-tensorflow) introduces TensorFlow through creative applications, including the histogram check where image-distribution mean over standard deviation should look sane. The notebooks that match a book-length version of that stack are [deep learning with keras](https://github.com/fchollet/deep-learning-with-python-notebooks), François Chollet's notebooks for Deep Learning with Python.

## Machine Learning Courses

Deep learning sits on a core ML course, not the other way around. [Recommended: Udacity includes ML and DL](https://classroom.udacity.com/courses/ud188/lessons/b4ca7aaa-b346-43b1-ae7d-20d27b2eab65/concepts/4b7026be-06e3-49de-a362-ce109172659e) is the Udacity course that includes both. The lecture pointers that used to sit under it, without their own addresses, are still the syllabus to walk: Week 1, Lesson 4, supervised and unsupervised; Lesson 6, model regression and the cost function; Lesson 71, the optimization objective and large-margin classification; PCA at Coursera in three parts; SVM at Coursera, the simplified first lecture.

## NLP Courses

After the core ML lectures, language is its own stack. [spacy](https://spacy.io/usage/spacy-101) is the 101, the concepts in spaCy's own terms. [gensim](https://www.machinelearningplus.com/nlp/gensim-tutorial/) is Selva Prabhakaran's tutorial for the library billed as topic modeling for humans. [2](https://radimrehurek.com/gensim/auto_examples/) is the official gensim examples. [nltk](https://realpython.com/nltk-nlp-python/) is the Real Python NLTK path. The second [2](https://www.tutorialspoint.com/natural_language_toolkit/index.htm) is TutorialsPoint's NLTK index, starting from language as a method of communication you can speak, read, and write. The [yandex](#life-cycle) pointer is an anchor back to the lifecycle section above rather than a separate course, and the Yandex course repo itself is in the notebooks below. [voita](https://lena-voita.github.io/nlp_course.html) is Lena Voita's course: interactive lectures, research exercises, and papers with summaries.

## Predictive Analytics Course

The last course on the shelf is predictive analytics as a lecture sequence, not another library tour. The [Syllabus](https://www.coursera.org/learn/predictive-analytics) is the Coursera course. Week 2 runs Lesson 29, supervised learning; Lesson 36, from rules to trees; Lesson 43, overfitting, then validation, then accuracy; Lesson 46, bootstrap, bagging, boosting, and random forests; Lesson 52, neural nets; Lesson 55, gradient descent; Lesson 59, logistic regression, SVM, regularization, lasso, and ridge; Lesson 64, gradient descent in stochastic, parallel, and batch forms. Unsupervised learning in that course is the lesson on k-means and DBSCAN.

## Books & notebooks

Courses end. The books are what you keep on the desk, and the design-pattern series is the one that connects training choices to production.

[Machine learning design patterns](https://www.oreilly.com/library/view/machine-learning-design/9781098115777/) is the O'Reilly book. The [git](https://github.com/GoogleCloudPlatform/ml-design-patterns) repo is the source that accompanies it. The [medium](https://lakshmanok.medium.com/machine-learning-design-patterns-58e6ecb013d7) notebooks are the walkthrough. The five patterns, in order, are the production habits the earlier lifecycle was pointing at: [DP1 - transform](https://medium.com/swlh/ml-design-pattern-1-transform-9e82ccbc3209) keeps inputs, features, and transforms separate so the model can move to production; [DP2 - checkpoints](https://medium.com/data-science/ml-design-pattern-2-checkpoints-e6ca25a4c5fe) saves intermediate weights for resilience, generalization, and tuning; [DP3 - virtual epochs](https://medium.com/google-cloud/ml-design-pattern-3-virtual-epochs-f842296de730) trains and evaluates on a total number of examples, not on epochs or steps; [DP4 - keyed predictions](https://medium.com/data-science/ml-design-pattern-4-keyed-predictions-a8de67d9c0f4) keys the predictions; [DP5 - repeatable sampling](https://medium.com/data-science/ml-design-pattern-5-repeatable-sampling-c0ccb2889f39) hashes a well-distributed column to split training, validation, and test.

Beside that series, [Gensim notebooks](https://github.com/RaRe-Technologies/gensim/tree/develop/docs/notebooks) run from word2vec and doc2vec through NMF, LDA, PCA, the scikit-learn API, cosine similarity, topic modeling, and t-SNE. [Deep learning with python](https://www.manning.com/books/deep-learning-with-python) is the Manning book for building applications with Python and Keras. The [git notebooks!](https://github.com/fchollet/deep-learning-with-python-notebooks) are François Chollet's notebooks for that book, covering deep learning and vision. The [official notebooks](https://github.com/PacktPublishing/Deep-Learning-with-Keras) are the Packt repo for Deep Learning with Keras. Yandex school's [nlp notebooks](https://github.com/yandexdataschool/nlp_course) are the course repo that matches the NLP shelf above. [Machine learning engineering book](http://www.mlebook.com/wiki/doku.php) is the companion wiki of Andriy Burkov's The Hundred-Page Machine Learning Book. [Interpretable Machine Learning book](https://christophm.github.io/interpretable-ml-book/) is Christoph Molnar's book. The same notes are in [Interpretable & Explainable AI (XAI)](/responsible-ai/interpretable-and-explainable-ai-xai).

## Cost

A trained model still has a bill. [GPT2/3](https://medium.com/modern-nlp/estimating-gpt3-api-cost-50282f869ab8) is how to estimate GPT API cost before the call volume surprises you.

## Patents

Cost is not the only constraint on what you ship. [Method Patent Exceptionalism](https://ilr.law.uiowa.edu/print/volume-102-issue-3/method-patent-exceptionalism) is the Iowa Law Review article on how method patents are treated differently, the legal boundary around a method you might think is just an algorithm.

## General Advice

After the role, the process, the courses, and the books, the last note is how to look at a large messy table without lying to yourself. [Practical advice for analysis of large, complex data sets](https://www.unofficialgoogledatascience.com/2016/10/practical-advice-for-analysis-of-large.html) covers distributions, outliers, examples, slices, whether a metric is significant, consistency over time, validation, description, evaluation, robustness of the measurement, and reproducibility.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Review of deep learning papers and co authorship. This address no longer opens: <https://neurovenge.antonomase.fr/>
* response 1. This address no longer opens: <https://www.highlights.lornerubis.com/page/83/>


# Next

Next is Data. A business question is answered with a dataset before a model is chosen, so once the metric the business wants changed is named, the next step is to find and shape the records that can move it.


# Introduction

The business problem is now a question about records, and the next work is to get those records into a form a model can use. After this part the reader can walk datasets, processing, analytics, and data engineering in that order.

The walk starts with the rows themselves, because a model is only as defined as the rows it is trained on: [Datasets](/data/datasets-intro) names feature types, finds a dataset, and judges whether the split and the label balance are fit to use. Those raw values are rarely on the scale or in the shape a learner expects, so [Data Processing](/data/processing-intro) scales, transforms, imputes, and annotates features and points at a pipeline that does it. Once the table is in shape, probability and dependence decide what a later score means, and [Analytics](/data/analytics-intro) reads distributions, dependence, and information measures so a feature is chosen for a reason. Finally the dataset has to live in a system other people can run, and [Data Engineering](/data/engineering-intro) places SQL, storage, quality, lineage, and ownership on one path from source to table.


# Datasets

A model is only as defined as the rows it is trained on. After this chapter the reader can name feature types, find a dataset, and judge whether the split and the label balance are fit to use.

The first question about a row is what each column is, and [Feature Types](/data/datasets-intro/feature-types) separates a label from an ordered rank from a real number, because encoding and regression treat those differently. Knowing the types is useless if nobody can find the data, so [Data Catalogs](/data/datasets-intro/data-catalogs) is the shelf that names what exists and the tools that implement one. A found dataset can still look good in training while its labels are noisy or its easy examples dominate, and [Dataset Confidence](/data/datasets-intro/dataset-confidence) uses dataset cartography to decide how much to trust it. With trust settled, [Datasets](/data/datasets-intro/datasets) is the work of typing the data, splitting it, and deciding when the sample is fair, from structured versus unstructured data through imbalance and transfer learning.


# Feature Types

A model needs to know whether a column is a label, an ordered rank, or a real number, because encoding and regression treat those differently. This page separates variable types, then discrete features, then continuous ones, and how categorical predictors become indicators. The same notes are in [Regression](/predictive-ml/regression).

The original Feature Types handout, a no permission doc, used to sit here; it no longer opens and is kept at the end of the page.

## Variable types

Before a column can be called discrete or continuous, it needs a measurement level. The [Variable types:](http://www.socialresearchmethods.net/kb/measlevl.php) note on levels of measurement is the reference for the three used here: nominal (weather), ordinal (order var 1,2,3), and interval (range).

## Discrete

The measurement levels split first into discrete features, which come as numbers or as categories. Categorical data are variables that contain label values rather than numeric values, and the number of possible values is often limited to a fixed set. Categorical variables are often called [nominal](https://en.wikipedia.org/wiki/Nominal_category), which is the Wikipedia entry on the nominal category. Those labels are usually discrete values such as gender, country of origin, marital status, or high-school graduate.

## Continuous

Continuous is the opposite of discrete: real-number values, measured on a continuous scale, such as height and weight. The two meet at the regression step. In order to compute a regression, categorical predictors must be re-expressed as numeric: some form of indicator variables (0/1) with a separate indicator for each level of the factor. The boundary also runs the other way, since discrete features with many values are often treated as continuous, i.e. zone numbers -> binary.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Feature Types - no permission doc. This address no longer opens: <http://www.biostat.umn.edu/\\~will/6470stuff/Class09-12/Handout09.pdf>


# Data Catalogs

Teams cannot use data they cannot find, so the catalog is the shelf that names what exists. This page is what a data catalog is, then the tools that implement one.

The same notes are in [Data Fabric](/data/engineering-intro/database-architecture-and-modeling#data-fabric) and [Data lineage vendors](/data/engineering-intro/data-lineage#data-lineage-vendors).

## Tools

The catalog idea becomes concrete in a product. [Alation](https://www.alation.com/) presents itself as the Alation Intelligence Operating System, which it says global enterprises trust to get their AI right and keep it right. Its own definition of the catalog is the one this page works from:

> A [data catalog](https://www.alation.com/product/data-catalog/) is a collection of metadata, combined with data management and search tools, that helps analysts and other data users to find the data that they need, serves as an inventory of available data, and provides information to evaluate fitness of data for intended uses.


# Dataset Confidence

A training run can look good while the labels are noisy or the easy examples dominate, so the question is how much to trust the dataset. This page is dataset cartography: mapping and diagnosing datasets with training dynamics.

The same notes are in [DISTILLING DATA](/data/datasets-intro/datasets#distilling-data).

The tool for that question is [**Dataset cartography mapping and diagnosing datasets with training dynamics -**](https://arxiv.org/abs/2009.10795) and its abstract starts from the same worry. Large datasets have become commonplace in NLP research. However, the increased emphasis on data quantity has made it challenging to assess the quality of data. We introduce Data Maps---a model-based tool to characterize and diagnose datasets. We leverage a largely ignored source of information: the behavior of the model on individual instances during training (training dynamics) for building data maps.

That behavior becomes something you can measure per row. This yields two intuitive measures for each example---the model's confidence in the true class, and the variability of this confidence across epochs---obtained in a single run of training. Experiments across four datasets show that these model-dependent measures reveal three distinct regions in the data map, each with pronounced characteristics.

Each region answers a different trust question. First, our data maps show the presence of "ambiguous" regions with respect to the model, which contribute the most towards out-of-distribution generalization. Second, the most populous regions in the data are "easy to learn" for the model, and play an important role in model optimization. Finally, data maps uncover a region with instances that the model finds "hard to learn"; these often correspond to labeling errors. Our results indicate that a shift in focus from quantity to quality of data could lead to robust models and improved out-of-distribution generalization.


# Datasets

A model is only as good as the dataset under it, so this page is the work of typing the data, splitting it, and deciding when the sample is fair. It moves from structured versus unstructured data through bias, sparsity, train and test splits, sample selection, imbalance, learning curves, distilling, selection, and transfer learning.

### Structured / Unstructured data

The first split is not train versus test but what kind of data you hold at all. The two Webopedia definitions, [Unstructured ](https://www.webopedia.com/TERM/U/unstructured_data.html)and [Structured](https://www.webopedia.com/TERM/S/structured_data.html), are the terms the rest of the page assumes.

### BIAS / VARIANCE

Once the data is typed, the next question is whether it misleads the model, and that starts with bias. The [Various Bias types](https://queue.acm.org/detail.cfm?id=3466134) piece from queue acm is the source of the bias taxonomy in the figure below.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2FxngZ8eUQ6iIfKPH6eDHB%2Fimage.png?alt=media&amp;token=78d17b97-bac7-40ed-8030-2011de884182" alt=""><figcaption><p>Various bias types.</p></figcaption></figure>

Bias also hides in how a test set is reused. Lauren Oakden-Rayner's "AI competitions don’t produce useful models" is the [Overfitting your test set, a statistican view point, a great article](https://lukeoakdenrayner.wordpress.com/2019/09/19/ai-competitions-dont-produce-useful-models/) link, the warning that a leaderboard can reward fitting the test set rather than the problem.

The working diagnosis comes from understanding what is the next stage in DL (& ML) algorithm development, the basic approach laid out by [Andrew NG](https://www.youtube.com/watch?v=F1ka6a13S9I). The terms are training, validation, and test. The split is training & validation 70%, test 30%. The procedure is cross fold training and validation, or a further split of the 70% into training and validation. With those three errors in hand, each situation reads off the gap between them.

BIAS, situation 1, is doing much worse than human. The human expert has 1% error, the training set error is 5% (test on train), and the validation set error is 6% (test on validation or CFV). The conclusion is that there is a BIAS between the human expert and the training set. The solution is 1. train deeper or bigger\larger networks, 2. train longer, 3. it may need more data to get to the human expert level, or 4. a new model architecture.

VARIANCE, situation 2, is a validation set error that is not close to the training set error. The human expert has 1% error, the training set error is 2%, and the validation set error is 6%. The conclusion is that there is a VARIANCE problem, i.e. OVERFITTING, between training and validation. The solution is 1. early stopping, 2. regularization, or 3. get more data, or 4. a new model architecture.

Situation 3 is both. The human expert has 1% error, the training set error is 5%, and the validation set error is 10%. The conclusion is that both problems occur, i.e., BIAS as well as VARIANCE, and the solution is to do it all.

The short version: underfitting means get more data; overfitting means early stop and regularization, because the model learns detail and noise. It happens more in non parametric (and non linear) algorithms such as decision trees. Bottom line, a bigger model or more data will solve most issues. The Stanford Machine Learning lecture 10, Advice for applying Machine Learning, gives that advice in practice with [regularized linear regression.](http://www.holehouse.org/mlclass/10_Advice_for_applying_machine_learning.html) The four figures below are the bias and variance situations drawn out.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-0f1e3c4be85da43a3942c5e96f22b4e18dff831f%2Fgimg-2ea62bce8904.png?alt=media" alt=""><figcaption><p>Bias and variance situations.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-09a8d8de9ebed52f4923cd3b4fcd3499c4cfc5d2%2Fgimg-1e8e213eb1ff.png?alt=media" alt=""><figcaption><p>Bias and variance situations.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-252a292b24871bc39081d951cd2ff3b569c67af1%2Fgimg-17fd5a5df7f1.png?alt=media" alt=""><figcaption><p>Bias and variance situations.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-f2102c574cfbd12aa873768a47443503791ce560%2Fgimg-874b14f8d11c.png?alt=media" alt=""><figcaption><p>Bias and variance situations.</p></figcaption></figure>

IMPORTANT! Those situations assume one distribution, and train/test efficiency changes when the data is from different distributions. For example, TRAIN is 50K hours of voice chatter as the train set for a DLN, and TEST is 10H for a specific voice-based problem, i.e., taxi chatter. Best practice is to divide the validation & test from the same distribution, i.e. the 10H set. The reason is that improving scores on a validation set from a different distribution will not be the same quality as improving scores on a validation set originated from the actual distribution of the problem’s data, i.e., 10H. NOTE: this is unlike the usual supervised learning, where all the data is from the same distribution and we split the training to train and validation (cfv).

[Situation 4](https://youtu.be/F1ka6a13S9I?t=47m26s) is the point in the same talk where this is resolved: however, when there are 2 distributions it’s possible to extend the division of the training set to validation\_training and training, and the test to validation and test. The split is Train, Valid\_Train = 48K\2K & Valid, Test, 5K & 5K, as drawn below.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-2e97a0f3261a1af6bfe4e271c5c67291f22944ef%2Fgimg-278a91013636.png?alt=media" alt=""><figcaption><p>Train and test when two distributions exist.</p></figcaption></figure>

So situation 1 stays the same. Situation 2 is the Valid\_Train error (train\_dev). Situation 3 is the Valid\_Test error, where the solution is more data, data synthesis that tweaks the test to be similar to the train data, or a new architecture. Situation 4 is now the Test set error, and the answer is to get more data.

### SPARSE DATASETS

Bias and variance are about the errors; sparsity is about the shape of the data that produces them. [Sparse matrices](https://machinelearningmastery.com/sparse-matrices-for-machine-learning/) in ML come from one hot/tfidf encodings and are stored as a dictionary/list of lists/ coordinate list.

###

### TRAINING METHODOLOGIES

With the data typed and its errors diagnosed, the next choice is how to train on it at all. The same notes are in [Fine tuning](/deep-learning/deep-neural-nets#fine-tuning), [Methods](/generative-ai/methods), and [Transfer Learning using CNN](/deep-learning/convolutional-nets#transfer-learning-using-cnn).

The methodologies run from the plain split to borrowing labels from another dataset or model:

1. Train test split
2. Cross validation
3. Transfer learning - using a pre existing classifier similar to your domain, usually trained on millions of samples. fine-tuned on new data in order to create a new classifier that utilizes that information in the new domain. Examples such as w2v or classic resnet fine-tuning.
4. Bootstrapping training- using a similar dataset, such as yelp, with 5 stars to create a pos/neg sentiment classifier based on 1 star and 5 stars. Finally using that to label or sample select from an unlabelled dataset, in order to create a new classifier or just to sample for annotation etc.
5. The [Student-teacher paradigm](https://developers.facebook.com/videos/2019/from-visual-recognition-to-reasoning/) (facebook), using a big labelled dataset to train a teacher classifier, predicting on unlabelled data, choosing the best classified examples based on probability, using those to train a new student model, finally fine-tune on the labeled dataset to create a more robust model, which is expected to know the unlabelled dataset and the labelled dataset with higher accuracy. With respect to the fully supervised teacher model / baseline.

The same notes are in [PRUNING / KNOWLEDGE DISTILLATION / LOTTERY TICKET](/deep-learning/deep-network-optimization#pruning--knowledge-distillation--lottery-ticket).

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-6375f4d6e46e237ed19fe3350802f67b29dc981d%2Fgimg-0cc4c443ecb9.png?alt=media" alt=""><figcaption><p>Student–teacher paradigm.</p></figcaption></figure>

6. Yoav’s method for transfer learning for languages - train a classifier on labelled data from english and spanish, fine tune using left out spanish data, stop before overfitting. This can be generalized to other domains.

### TRAIN / TEST / CROSS VALIDATION

The first two methodologies above, the split and cross validation, need more care than a single line. The same notes are in [STATISTICAL SAMPLING AND RESAMPLING](/data/analytics-intro/probability-and-statistics#statistical-sampling-and-resampling).

When rows belong to groups, Scikit-lego on group-based splitting and transformation is the tool for splitting by group; its archived page is in the reference list under dataset selection below.

The holdout and cross-validation figures below are credited to [Images from here](https://www.kdnuggets.com/2017/08/dataiku-predictive-model-holdout-cross-validation.html), the KDnuggets piece on making predictive models robust. It explains that the validation step helps you find the best parameters for your model and prevent overfitting, and weighs the pros and cons of the hold-out strategy against k-fold.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-6740ae739feade283f49dddb93e14973f5f0ff8f%2Fgimg-543343a4119e.png?alt=media" alt=""><figcaption><p>Holdout and cross-validation.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-2ece367b7fc878514065fc2309e1c87825463079%2Fgimg-1e1a60204df9.png?alt=media" alt=""><figcaption><p>Holdout and cross-validation.</p></figcaption></figure>

Jason Brownlee's [Train Test methodology](http://machinelearningmastery.com/how-to-choose-the-right-test-options-when-evaluating-machine-learning-algorithms/) post is about choosing test options, which can mean the difference between over-learning, a mediocre result, and a usable state-of-the-art result, and it walks the standard options below. Before that, the roles of the three sets need to be fixed, and the Stack Exchange answer on test versus validation sets puts it this way:

“[The training](https://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set) set is used to fit the models; the validation set is used to estimate prediction error for model selection; the test set is used for assessment of the generalization error of the final chosen model. Ideally, the test set should be kept in a “vault,” and be brought out only at the end of the data analysis”

The test options then build on each other, each fixing the previous one's problem. Random split tests of 66\33 have a problem: variance each time we rerun. Multiple random split tests have their own problem: samples may not be included in train\test or may be selected multiple times. Cross validation is pretty good, though a different random seed results in a different mean accuracy, with variance due to randomness. Multiple cross validation accounts for the randomness of the CV. Finally, statistical significance (t-test) on multi CV asks whether two samples are drawn from the same population (no difference); if “yes”, the result is not significant, even if the mean and std deviations differ.

Finally, when in doubt, use k-fold cross validation (k=10) and use multiple runs of k-fold cross validation with statistical significance tests. For ensembles there is one more variant: [Out of fold](https://machinelearningmastery.com/out-of-fold-predictions-in-machine-learning/) predictions - leave unseen data, do cross fold on that. Good for ensembles.

### SAMPLE SELECTION

A split only divides the rows you already have; sample selection decides how many rows you need in the first place. The same notes are in [A/B Testing](/decision-intelligence/a-b-testing).

The guide on [How to choose your sample size from a population based on confidence interval](https://www.checkmarket.com/blog/how-to-estimate-your-population-and-survey-sample-size/) now lands on Medallia Agile Research, the Medallia page for DIY surveys and research analytics. The figure below is the sample size from a population picture that sat with that guide.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-575c44b2df7c67be4931bca08d815397e180b90f%2Fgimg-02ac465d3915.png?alt=media" alt=""><figcaption><p>Sample size from a population.</p></figcaption></figure>

For training data rather than surveys, MachineLearningMastery's "How Much Training Data is Required for Machine Learning?" is the [Data advice, should we get more data? How much](https://machinelearningmastery.com/much-training-data-required-machine-learning/) question answered directly.

Sometimes the problem is not how many samples but how to draw them at all. Gibbs sampling: Gibbs Sampling is a MCMC method to draw samples from a potentially really really complicated, high dimensional distribution, where analytically, it’s hard to draw samples from it. The usual suspect would be those nasty integrals when computing the normalizing constant of the distribution, especially in Bayesian inference. Now Gibbs Sampler can draw samples from any distribution, provided you can provide all of the conditional distributions of the joint distribution analytically. A Python implementation is in the reference list under dataset selection below.

###

### IMBALANCED DATASETS

Even a well-sized sample can be lopsided, and imbalance is the case where one label dominates. The same notes are in [Decision Trees](/predictive-ml/decision-trees) and [Unbalanced labels](/problem-framing/label-algorithms#unbalanced-labels).

The imbalanced-learn over-sampling guide is the place to start ([the BEST resource and a great api for python)](http://contrib.scikit-learn.org/imbalanced-learn/stable/over_sampling.html) with visual samples - it actually works well on clustering. MachineLearningMastery then covers the two main levers: Cost-Sensitive Learning for Imbalanced Classification is the [Mastery on](https://machinelearningmastery.com/cost-sensitive-learning-for-imbalanced-classification/) cost-sensitive link, and SMOTE for Imbalanced Classification with Python is the [Smote for imbalance](https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/) walkthrough.

For deep networks the evidence is the [Systematic Investigation of imbalance effects in CNN’s](https://arxiv.org/abs/1710.05381), with several observations. This is crucial when training networks, because in real life you don’t always get a balanced DS.

They recommend the following:

1. (i) the effect of class imbalance on classification performance is detrimental;
2. (ii) the method of addressing class imbalance that emerged as dominant in almost all analyzed scenarios was oversampling;
3. (iii) oversampling should be applied to the level that totally eliminates the imbalance, whereas undersampling can perform better when the imbalance is only removed to some extent;
4. (iv) as opposed to some classical machine learning models, oversampling does not necessarily cause overfitting of CNNs;
5. (v) thresholding should be applied to compensate for prior class probabilities when overall number of properly classified cases is of interest.

Those findings reduce to a few general rules:

1. Many samples - undersampling
2. Few samples - over sampling
3. Consider random and non-random schemes
4. Different sample rations, instead of 1:1 (proof? papers?)

The full menu of balancing techniques follows. The Wikipedia entry on oversampling and undersampling in data analysis, the imbalanced-learn package for tackling the curse of imbalanced datasets, and its documentation examples are the references for balancing data sets ([wiki](https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis), [scikit learn](https://github.com/scikit-learn-contrib/imbalanced-learn) & [examples in SKLEARN](http://contrib.scikit-learn.org/imbalanced-learn/auto_examples/index.html)):

1. Oversampling the minority class

* (Random) duplication of samples
* SMOTE (in weka + needs to be installed & paper) - find k nearest neighbours,

$$\text{New\_Sample} = (\text{random num in \[0,1]}) \* \text{vec(ki,current\_sample)}$$

* (in weka) The nearestNeighbors parameter says how many nearest neighbor instances (surrounding the currently considered instance) are used to build an in between synthetic instance. The default value is 5. Thus the attributes of 5 nearest neighbors of a real existing instance are used to compute a new synthetic one.
* (in weka) The percentage parameter says how many synthetic instances are created based on the number of the class with less instances (by default - you can also use the majority class by setting the -Coption). The default value is 100. This means if you have 25 instances in your minority class, again 25 instances are created synthetically from these (using their nearest neighbours' values). With 200% 50 synthetic instances are created and so on.
* ADASYN - shifts the classification boundary to the minority class, synthetic data generated for majority class.

2. Undersampling the majority class

* Remove samples
* Cluster centroids - replaces a cluster of samples (k-means) with a centroid.
* Tomek links - cleans overlapping samples between classes in the majority class.
* Penalizing the majority class during training

3. Combined over and under (hybrid) - i.e., SMOTE and tomek/ENN
4. Ensemble sampling

* EasyEnsemble
* BalanceCascade

5. Dont balance, try algorithms that perform well with unbalanced DS

* Decision trees - C4.5\5\CART\Random Forest
* SVM

6. Penalize Models -

* added costs for misclassification on the minority class during training such as penalized-SVM
* a [CostSensitiveClassifier](http://weka.sourceforge.net/doc.dev/weka/classifiers/meta/CostSensitiveClassifier.html) meta classifier in Weka that wraps classifiers and applies a custom penalty matrix for miss classification.
* complex

##

### LEARNING CURVES

Every fix above, from more data to rebalancing, needs a way to see whether it helped, and learning curves are that view. The [Git examples](https://gist.github.com/orico/260097cb1a2926c6b6ca6f71c37c135b) gist is Learning Curves for under/overfitting evaluation, and the [Sklearn examples](https://stats.stackexchange.com/questions/283738/sklearn-learning-curve-example) thread starts from scikit-learn's learning-curve figure, where an SVM seems to need more training examples, and asks how that is justified when both the training and validation scores are near perfect. For the bias and variance reading, the Practical advice for applying machine learning post on a technology, programming, and machine learning blog is the [Understanding bias variance via learning curves](http://digitheadslabnotebook.blogspot.com/2011/12/practical-advice-for-applying-machine.html) link.

Curves can also decide how much to sample. [learning curve sampling applied to model based clustering](http://www.jmlr.org/papers/volume2/meek02a/meek02a.pdf) - seemed like active learning, i.e., sample using EM/cluster to achieve nearly as accurate on all data. That leads to predicting the sample size required for training, and the morning paper's "Deep learning scaling is predictable, empirically" is the [Advice on many things, including learning curves](https://blog.acolyer.org/2018/03/28/deep-learning-scaling-is-predictable-empirically/amp/) link:

This is a really wonderful study with far-reaching implications that could even impact company strategies in some cases. It starts with a simple question: “how can we improve the state of the art in deep learning?” We have three main lines of attack:

1. We can search for improved model architectures.
2. We can scale computation
3. We can create larger training data sets.

### DISTILLING DATA

Larger training sets are one answer; distilling asks whether a smaller, better set does as well. The same notes are in [Dataset Confidence](/data/datasets-intro/dataset-confidence).

The Medium write-up on this used to sit here and is kept at the end of the page; the paper itself is [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://arxiv.org/abs/2009.10795). What I found interesting about this paper is that it challenges the common approach of “the more the merrier” when it comes to training data, and shifts the focus from the quantity of the data to the quality of the data.

###

### DATASET SELECTION

After deciding which rows to keep, the last choice is which dataset, or which source model, to start from. The same notes are in [Fine tuning](/deep-learning/deep-neural-nets#fine-tuning), [Methods](/generative-ai/methods), and [Transfer Learning using CNN](/deep-learning/convolutional-nets#transfer-learning-using-cnn).

"How to Choose the Best Source Model for Transfer Learning" on [Medium](https://medium.com/@amielmeiseles/how-to-choose-the-best-source-model-for-transfer-learning-41d5c91c1338) assumes a basic familiarity with transfer learning and points readers who need a refresher to A Comprehensive Hands-on Guide to Transfer Learning with Real-World Applications in Deep Learning; its figure is below.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-2d6a0ebb9f405bee230b4089c77e8d1c5c867087%2Fgimg-3332b5be0d60.png?alt=media" alt=""><figcaption><p>Dataset selection for transfer learning.</p></figcaption></figure>

The full addresses of the sources used across this page are collected here, each tied to the section it serves:

* The test-set overfitting warning from the bias section, Overfitting your test set, a statistican view point, a great article: <https://lukeoakdenrayner.wordpress.com/2019/09/19/ai-competitions-dont-produce-useful-models/?fbclid=IwAR1WM5U7imq-2LFPifyCoTPp-MFwPoGROMLr2TZWAp41qgVeLdT-2bkLyk&blogsub=confirming#subscribe-blog>
* Cost-Sensitive Learning for Imbalanced Classification - MachineLearningMastery.com, the Mastery on cost sensitive sampling from the imbalance section: <https://machinelearningmastery.com/cost-sensitive-learning-for-imbalanced-classification/?fbclid=IwAR0_DeIydTAAkutypcMBfrnC4QyuyqVxDu_uej5t48AvQKShcRUqfMm8Rqo>
* SMOTE for Imbalanced Classification with Python - MachineLearningMastery.com, the Smote for imbalance walkthrough: <https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/?fbclid=IwAR3W59c54ohoaIHnHLQFCcZZanFXI4QzIzuWiUtaUC851JFkwlevCAgvpbM>
* The paper behind the distilling section, Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics: <https://arxiv.org/abs/2009.10795>
* A Comprehensive Hands-on Guide to Transfer Learning with Real-World Applications in Deep Learning, subtitled Deep Learning on Steroids with the Power of Knowledge Transfer!, for transfer learning in deep learning: <https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a>
* The archived Scikit-lego page on group-based splitting and transformation from the train/test section: <https://web.archive.org/web/2020/https://scikit-lego.readthedocs.io/en/latest/meta.html#Grouped-Prediction>
* Agustinus Kristiadi's blog example of a Gibbs sampling implementation in Python to sample from a Bivariate Gaussian, for the Gibbs sampling note: <https://web.archive.org/web/2020/https://wiseodd.github.io/techblog/2015/10/09/gibbs-sampling/>

#### TRANSFER LEARNING

A chosen source model ends the story where the methodologies began, with transfer learning, drawn here as it is done in deep learning.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-3fc10fe0c9dc743c50a07456ced196a72159788f%2Fgimg-1254419245f4.png?alt=media" alt=""><figcaption><p>Transfer learning.</p></figcaption></figure>

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

\- Scikit-lego on group-based splitting and transformation. This address no longer opens: <https://scikit-lego.readthedocs.io/en/latest/meta.html#Grouped-Prediction> - SMOTE (in weka + needs to be installed & paper). This address no longer opens: <http://www.jair.org/media/953/live-953-2037-jair.pdf> - Gibbs sampling:. This address no longer opens: <https://wiseodd.github.io/techblog/2015/10/09/gibbs-sampling/> - Advice on many things, including learning curves. This address no longer opens: <https://blog.acolyer.org/2018/03/28/deep-learning-scaling-is-predictable-empirically/amp/?fbclid=IwAR0V1X1vuCZYmeku12YHJI7wwK7RCKEyE2Q7aRDDT58hjRPzAOrHfvo98WY> - Medium on this Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. This address no longer opens: <https://towardsdatascience.com/data-maps-datasets-can-be-distilled-too-1991c3c260d6>


# Named Datasets

Practice and baselines need public hubs, not only private tables. This page is the hub list for public datasets and models, first for practice projects, then for translation, then for pre-trained models.

### VARIOUS DATASETS

The quickest way to practice is a project that already comes with its data. [26 of them](https://www.analyticsvidhya.com/blog/2018/05/24-ultimate-data-science-projects-to-boost-your-knowledge-and-skills/?utm_source=facebook.com\&utm_medium=social) is Analytics Vidhya's list of machine learning projects for beginners, meant for learning data science by applying it and showing the skills on a CV.

Translation needs paired text rather than a single table. [24](https://lionbridge.ai/datasets/25-best-parallel-text-datasets-for-machine-translation-training/) is the Lionbridge list of parallel text datasets for machine translation training. For one concrete pair, English-Spanish, [Eu-](https://datarepository.wolframcloud.com/resources/Europarl-English-Spanish-Machine-Translation-Dataset-V7)es is the Wolfram Data Repository's Europarl V7, a parallel corpus for machine translation from the proceedings of the European Parliament, and [2](https://data.europa.eu/euodp/en/data/dataset/elrc_339) is the same kind of resource on the European Data Portal.

When the baseline is a model rather than a dataset, the hub is 50K models deep: [ModelDepot](https://modeldepot.io/) alone has over 50,000 freely accessible pre-trained models with search functionality to find them.


# Next

Next is Data Processing. A dataset whose rows are typed, found, trusted, and split still has to be transformed before it is modeled, because raw values are rarely on the scale or in the shape a learner expects.


# Data Processing

Raw values are rarely on the scale or in the shape a learner expects. After this chapter the reader can scale, transform, impute, and annotate features, and can point at a pipeline that does it.

The chapter starts with shape, because skewed data breaks tests that assume a normal shape: [Distribution Transformation](/data/processing-intro/distribution-transformation) transforms the column, mainly with Box-Cox, and then checks whether normality actually arrived. Shape is only half of it, since features on different scales pull a model toward the loud columns, and [Normalization & Scaling](/data/processing-intro/normalization-and-scaling) puts them on a shared scale without inventing signal. Those steps have to run somewhere repeatable, so [Data Pipelines](/data/processing-intro/data-pipelines) moves data from source systems into a place a model or warehouse can use and asks which integration path and orchestrator own that path. With clean columns flowing, [Feature Engineering](/data/processing-intro/feature-engineering) builds features, fills gaps, and measures how alike two rows are. The last input is the label itself, and [Annotation & Disagreement](/data/processing-intro/annotation-and-disagreement) treats labels as work product that annotators disagree on, from myths and agreement metrics to the tools that run that loop.

To see this step on a real table, [Cleaning Data With AI Denoisers](https://pub.towardsai.net/cleaning-data-with-ai-denoisers-be1bdea0fe20) (October 2024) shows this step on a real table: it presents AI denoisers as algorithms that reduce noise in data such as images, audio, and video, where traditional denoising relies on filters and statistical methods.


# Distribution Transformation

Skewed data breaks tests that assume a normal shape, so the work here is to transform the column and then check whether normality actually arrived. Box-Cox is the main transform on this page; guaranteed normality is the honest answer, then the null-hypothesis tests and the Mann-Whitney U test that do not need normality. The same notes are in [CORRELATION](/data/analytics-intro/dependence-and-selection#correlation), [Distribution](/data/analytics-intro/distribution), and [FEATURE SELECTION](/data/analytics-intro/dependence-and-selection#feature-selection).

The usual fixes for a skewed column are few, and the [Top 3 methods for handling skewed data](https://medium.com/data-science/top-3-methods-for-handling-skewed-data-1334e0debf45) are log, square root, and box cox transformations. The last of these gets the rest of the page.

## Box Cox

Of those three, Box-Cox is the one that picks its own exponent. The same notes are in [FEATURE ENGINEERING](/data/processing-intro/feature-engineering#feature-engineering).

Jason Brownlee's [Power transformations](https://machinelearningmastery.com/power-transforms-with-scikit-learn/) post starts from the same problem: algorithms like Linear Regression and Gaussian Naive Bayes assume numerical variables have a Gaussian distribution, while real data may be nearly Gaussian with outliers or a skew, or a totally different distribution such as exponential. So what is the Box-Cox Power Transformation? It is a procedure to identify an appropriate exponent (Lambda = l) to use to transform data into a “normal shape.” The Lambda value indicates the power to which all data should be raised.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-97e64e9f61553f51516ae321e4316564dd1896f4%2Fgimg-61c2ca6ced35.png?alt=media" alt=""><figcaption><p>Box-Cox power transformation.</p></figcaption></figure>

The NIST handbook's Box-Cox Normality Plot page explains why that matters, and opens with [The Box-Cox transformation is a useful family of transformations.](http://www.itl.nist.gov/div898/handbook/eda/section3/eda336.htm) Many statistical tests and intervals are based on the assumption of normality. The assumption of normality often leads to tests that are simple, mathematically tractable, and powerful compared to tests that do not make the normality assumption. Unfortunately, many real data sets are in fact not approximately normal. However, an appropriate transformation of a data set can often yield a data set that does follow approximately a normal distribution, which increases the applicability and usefulness of statistical techniques based on the normality assumption.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-ef32e232aee282481cfd69d217a7eaf1e2e73bf4%2Fgimg-a3c10a4ae254.png?alt=media" alt=""><figcaption><p>An appropriate transformation can yield an approximately normal distribution.</p></figcaption></figure>

IMPORTANT:!! After a transformation (c), we need to measure the normality of the resulting transformation (d). One measure is to compute the correlation coefficient of a [normal probability plot](http://www.itl.nist.gov/div898/handbook/eda/section3/normprpl.htm) => (d). The correlation is computed between the vertical and horizontal axis variables of the probability plot and is a convenient measure of the linearity of the probability plot. In other words: the more linear the probability plot, the better a normal distribution fits the data!

Minitab's guest post, How Could You Benefit from a Box-Cox Transformation?, is the [\*NOTE: another useful link that explains it with figures, but i did not read it.](http://blog.minitab.com/blog/applying-statistics-in-quality-projects/how-could-you-benefit-from-a-box-cox-transformation)

## Guaranteed normality?

That normality check is not optional, because Box-Cox does not guarantee a normal result. The answer is NO! This is because it actually does not really check for normality; the method checks for the smallest standard deviation. The assumption is that among all transformations with Lambda values between -5 and +5, transformed data has the highest likelihood – but not a guarantee – to be normally distributed when standard deviation is the smallest. It is absolutely necessary to always check the transformed data for normality using a probability plot. (d)

Additionally, the Box-Cox Power transformation only works if all the data is positive and greater than 0. This is achieved easily by adding a constant ‘c’ to all data such that it all becomes positive before it is transformed. The transformation equation is then given by the Statistics How To definition page, [COMMON TRANS: FORMULAS (based on the actual formula)](http://www.statisticshowto.com/box-cox-transformation/), shown in the figure below.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-b5171cc1c03680f9a664f187d3e1d38429ac60a2%2Fgimg-24c7deea825a.png?alt=media" alt=""><figcaption><p>Common transformation formulas.</p></figcaption></figure>

Finally: An awesome tutorial (dead) used to explain this, and a replacement is kept at the end of the page. In python there are [code examples](https://github.com/kentmacdonald2/Box-Cox-Transformation-Python-Example), the companion code from the Box-Cox transformation tutorial on kmdatascience.com, and there is also another code example [here](https://stackoverflow.com/questions/33944129/python-library-for-data-scaling-centering-and-box-cox-transformation), the Stack Overflow question asking for a Python package that does scaling, centering, and Box-Cox to eliminate skewness, the way R's caret package does. The quote below explains what that function call returns:

> “Simply pass a 1-D array into the function and it will return the Box-Cox transformed array and the optimal value for lambda. You can also specify a number, alpha, which calculates the confidence interval for that value. (For example, alpha = 0.05 gives the 95% confidence interval).”

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-82035048a997ef6f89320be5714c6445c7346223%2Fgimg-f209afaf26e3.png?alt=media" alt=""><figcaption><p>Box-Cox transformed array and lambda.</p></figcaption></figure>

Maybe there is a slight problem in the python vs R code; the details here are kept at the end of the page, but it needs investigating.

## Null hypothesis

When normality cannot be guaranteed, the result still has to be tested against a null hypothesis. The same notes are in [Hypothesis Testing](/decision-intelligence/hypothesis-testing).

A note on chi-square, the null hypothesis, and observed versus expected counts used to sit here; that address no longer opens and is kept at the end of the page.

Analytics vidhya covers the rest of the path. [What is hypothesis testing](https://www.analyticsvidhya.com/blog/2015/09/hypothesis-testing-explained/) is Sunil Ray's guide: hypothesis testing is a data analysis technique used to make inferences about the sample data from a larger population. [Intro to t-tests analytics vidhya](https://www.analyticsvidhya.com/blog/2019/05/statistics-t-test-introduction-r-implementation/?utm_source=facebook.com\&utm_medium=social) teaches T-Tests, their types, uses, and formulas with R examples, and how they differ from ANOVA and when to perform them. [Anova analysis of variance](https://www.analyticsvidhya.com/blog/2018/01/anova-analysis-of-variance/?utm_source=facebook.com\&utm_medium=social) is Gurchetan's ANOVA basics: concepts, Excel usage, key terms, group variability, One-Way & Two-Way ANOVA, F-Statistic, and MANOVA explained with steps & examples.

The ANOVA piece answers whether the means of two or more groups are significantly different from each other. ANOVA checks the impact of one or more factors by comparing the means of different samples. A one-way ANOVA tells us that at least two groups are different from each other, but it won’t tell us which groups are different. When the outcome or dependent variable (in our case the test scores) is affected by two independent variables/factors, we use a slightly modified technique called two-way ANOVA. The multivariate case, and the technique we will use to solve it, is known as MANOVA.

The Box-Cox tutorial that was once marked as a tutorial (dead) is live again as Box Cox Transformations in Python, which starts from the point that many common machine learning algorithms assume data is normally distributed and asks what to do when it isn't: <http://www.kmdatascience.com/2017/07/box-cox-transformations-in-python.html>

## Mann-Whitney U test

If the data will not become normal at all, the test itself has to drop the assumption. The same notes are in [Hypothesis Testing](/decision-intelligence/hypothesis-testing).

The Wikipedia entry ([what is?](https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test)) defines it: the Mann–Whitney U test is a [nonparametric](https://en.wikipedia.org/wiki/Nonparametric_statistics) [test](https://en.wikipedia.org/wiki/Statistical_hypothesis_test) of the [null hypothesis](https://en.wikipedia.org/wiki/Null_hypothesis) that it is equally likely that a randomly selected value from one sample will be less than or greater than a randomly selected value from a second sample.

In other words: This test can be used to determine whether two independent samples were selected from populations having the same distribution.

Unlike the [t-test](https://en.wikipedia.org/wiki/T-test) it does not require the assumption of [normal distributions](https://en.wikipedia.org/wiki/Normal_distribution). It is nearly as efficient as the t-test on normal distributions.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* here is a new one. This address no longer opens: <https://towardsdatascience.com/box-cox-transformation-explained-51d745e34203>
* details here. This address no longer opens: <http://shahramabyari.com/2015/12/21/data-preparation-for-predictive-modeling-resolving-skewness/>
* What is chi-square and what is a null hypothesis, and how do we calculate observed vs expected and check if we can reject the null and get significant difference. This address no longer opens: <https://medium.com/greyatom/goodness-of-fit-using-chi-square-be5bba375caf>


# Normalization & Scaling

Features on different scales pull a model toward the loud columns, so the work is to put them on a shared scale without inventing signal. This page compares normalization and scaling choices, including when to standardize versus normalize.

The same notes are in [BATCH NORMALIZATION](/deep-learning/deep-neural-nets#batch-normalization), [Data Normalization/Augmentation](/deep-learning/deep-network-optimization#data-normalizationaugmentation), [KNN](/predictive-ml/nearest-neighbors#knn), [LDA - Linear discriminant analysis](/predictive-ml/dimensionality-reduction-methods#lda---linear-discriminant-analysis), and [Regression](/predictive-ml/regression).

The problem is easiest to see on real columns. [A comparison of normalization / scaling techniques in sklearn](http://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py) compares the effect of different scalers on data with outliers, using two California Housing features, median income in a block and average house occupancy, that have very different scales and contain some very large outliers. Ben Alex Keen's Feature Scaling with scikit-learn is [Another great explanation on sklearn and (general) scaling](http://benalexkeen.com/feature-scaling-with-scikit-learn/) - normal, min max, etc. Jason Brownlee's [Normalization/standardize features](http://machinelearningmastery.com/normalize-standardize-machine-learning-data-weka/) does the same in Weka, starting from the point that raw data often has attributes with varying scales, one in kilograms and another a count, and that carefully choosing a rescaling method can often give a boost in performance.

Because data has varying scales, the choice is between two targets. Normalize to a range of 0 to 1 when the algorithm you are using does not make assumptions about the distribution of your data, such as k-nearest neighbors and artificial neural networks. Standardize to a mean of 0 and a std of 1 when the algorithm assumes a Gaussian dist, such as linear regression, logistic regression and linear discriminant analysis (LR, LogR, LDA).

Generally, it is a good idea to standardize data that has a Gaussian (bell curve) distribution and normalize otherwise. In general terms, we should test 0,1 or -1,1 empirically and possibly match the range to the NN gates/activation function etc.


# Data Pipelines

Data has to move from source systems into a place a model or warehouse can use, and the choice is which integration path and which orchestrator owns that path. This page starts with the pipeline and integration notes, then Streaming vs Batch, Airflow, and CDC. The same notes are in [Airflow](/ai-engineering/full-stack-and-ops#airflow) and [General](/data/engineering-intro/data-engineering-questions#general).

The first choice is the integration tool, and the market is crowded. Metaplane's [Decision guide for data integration tools](https://www.metaplane.dev/blog/decision-guide-to-choosing-a-data-integration-tool) is the guide for choosing the right solution in it.

Before picking a tool, the order of the steps matters. An ETL pipeline is a type of data pipeline in which a set of processes extracts data from one system, transforms it, and loads it into a target repository; Qlik's [ETL vs ELT](https://www.qlik.com/us/etl/etl-vs-elt) compares that with a general data pipeline, and Guru99's ETL vs ELT – Difference Between Them is the [2](https://www.guru99.com/etl-vs-elt.html) second explanation. The data engineering industry has evolved from Extract, Transform, Load (ETL) to ELT, where raw data is copied from the source system and loaded into a data store first, and [What is reverse ETL (ELT)](https://torbjornzetterlund.com/what-is-reverse-etl/) picks up from that shift.

With the pattern chosen, the tools are the ones that carry data in. [Airbyte](https://airbyte.com/) — open-source ELT data integration for modern data teams. Apache Airflow, Rivery and Stitch are all popular ETL tools for data ingestion into cloud data warehouses, and [Stitch vs Airflow vs Rivery](https://www.stitchdata.com/vs/airflow/rivery/) is the quick guide comparing their features, pricing, and services. Rivery and Fivetran are the other names on the shelf. Fivetran connects to Airflow directly: a new Python package from Fivetran and Astronomer enables connector management in Airflow, announced as [Operators for AirFlow](https://www.fivetran.com/blog/announcing-the-fivetran-airflow-provider). Astronomer's own guide to the best ETL tools to integrate with Airflow, on how to select one and why it pays to integrate Fivetran, Airbyte, and Azure Data Factory with Apache Airflow®, is [Astronomer AirFlow as an orchestrator scheduler for FT, AB, etc](https://www.astronomer.io/blog/best-etl-tools-airflow/).

A tool only helps if the pipeline stays reliable. [The 7 Principles of reliable data pipelines](https://medium.com/bigeye/seven-principles-for-reliable-data-pipelines-e82a82810e4f) Bigeye Medium article opens with the familiar pressure: “Yea when can you have that delivered by? We wanna do the science part soon.” For teams that want no code at all, [Hevo](https://hevodata.com/integrations/pipeline/) — Hevo is a Fully Automated, No-code Data Pipeline Platform that supports 150+ ready-to-use integrations across Databases, SaaS Applications, Cloud Storage, SDKs, and Streaming Services.

## Streaming vs Batch

Every one of those pipelines still has to decide how often data moves. The question to be able to answer is this: explain the difference and the reason to choose using Streaming over Batch and vice versa. Give an example for a project where you had to make this choice, and walk through your reasoning.

## Airflow

Whichever cadence wins, something has to schedule it, and Airflow is the orchestrator named throughout the tool shelf above. The same notes are in [Airflow](/ai-engineering/full-stack-and-ops#airflow).

The questions to be able to answer about it are:

1. What is Airflow?
2. How do you transfer information between tasks in Airflow?
3. Please give me a real-world example of using Spark and Airflow together

## CDC

An orchestrator moves whole batches; change data capture moves only what changed. The same notes are in [Tools](/data/engineering-intro/tools).

[What is a CDC and why do you need it, or how do you use it?](https://rockset.com/blog/change-data-capture-what-it-is-and-how-to-use-it/) — Change data capture (CDC) is the process of recognising when data has been changed in a source system so a downstream process or system can action. A common use case is to reflect (replication) the change in a different target system so that the data in the systems stay in sync.


# Feature Engineering

Raw columns are rarely what a model needs, so this page is the work of building features, filling gaps, and measuring how alike two rows are. It moves from feature engineering and imputing, through representation learning, into similarity, distance, and the distance tools.

### **FEATURE ENGINEERING**

Building a feature starts with looking at the data, and then with one guide per data type: numbers, categories, and text. The same notes are in [Auto Feature Engineering](/deep-learning/meta-learning#auto-feature-engineering), [Box Cox](/data/processing-intro/distribution-transformation#box-cox), and [Feature engineering](/data/engineering-intro/lakes-and-warehouses#feature-engineering).

Sunil Ray's comprehensive guide to data exploration is the starting point, a tutorial that comprises missing value imputation, outliers, feature engineering, and variable creation: [**Vidhya on FE, anomalies, engineering, imputing**](https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/?utm_source=outlierdetectionpyod\&utm_medium=blog).

From there, Dipanjan's Understanding Feature Engineering series takes one data type per part. Part 1, Continuous Numeric Data, starts from the idea that data makes the world go round and is [**Many types of FE**](https://medium.com/data-science/understanding-feature-engineering-part-1-continuous-numeric-data-da4e47099a7b)**, including log and box cox transform - a very useful explanation.** Part 2 moves from continuous numbers to categorical data, the discrete kind of structured data: [**Categorical Data**](https://medium.com/data-science/understanding-feature-engineering-part-2-categorical-data-f54324193e63), where the hashing is really cool, and the same article is the [**Dummy variables and feature hashing**](https://medium.com/data-science/understanding-feature-engineering-part-2-categorical-data-f54324193e63) reference. Part 3, Traditional Methods for Text Data, is [**Text data**](https://medium.com/data-science/understanding-feature-engineering-part-3-traditional-methods-for-text-data-f6f7d70acd41) **- unigrams, bag of words, N-grams (2,3,..), tfidf matrix, cosine\_similarity(tfidf) ontop of a tfidf matrix, unsupervised hierarchical clustering with similarity measures on top of (cosine\_similarity), LDA for topic modelling in sklearn - pretty awesome, Kmeans(lda),.** That is Dipanjan feature engineering across cont numeric, categorical, and traditional methods.

Part 4 is the hands-on, intuitive approach to deep learning methods for text data (Word2Vec, GloVe and FastText), for turning noisy, unstructured text into structured, vectorized formats any machine learning algorithm can use. It is both [**Deep learning data for FE**](https://medium.com/data-science/understanding-feature-engineering-part-4-deep-learning-methods-for-text-data-96c44370bbfa) and [**Word embedding using keras, continuous BOW - CBOW, SKIPGRAM, word2vec - really good.**](https://medium.com/data-science/understanding-feature-engineering-part-4-deep-learning-methods-for-text-data-96c44370bbfa) Christine Doig's PyGotham 2015 Introduction to Topic Modeling in Python is [**Topic Modelling**](http://chdoig.github.io/pygotham-topic-modeling/#/) **- a fantastic slide show about topic modelling using LDA etc.**

The same notes are in [LDA (Latent Dirichlet Allocation)](/natural-language-processing-nlp/topics-modeling#lda-latent-dirichlet-allocation).

For quick reference, **Dipanjan on feature engineering** is parts [**1**](https://medium.com/data-science/understanding-feature-engineering-part-1-continuous-numeric-data-da4e47099a7b) **- cont numeric**, [**2 -**](https://medium.com/data-science/understanding-feature-engineering-part-2-categorical-data-f54324193e63) **categorical**, and [**3**](https://medium.com/data-science/understanding-feature-engineering-part-3-traditional-methods-for-text-data-f6f7d70acd41) for text. Categories also need encoders beyond dummies, and the two Python packages for that are [**Target encoding git**](https://pypi.org/project/target_encoding/) and [**Category encoding git**](https://pypi.org/project/category-encoders/).

### **FEATURE IMPUTING**

A built feature is only usable if its gaps are filled. The same data exploration guide by Sunil Ray covers missing value imputation alongside outliers and variable creation: [**Vidhya on FE, anomalies, engineering, imputing**](https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/?utm_source=outlierdetectionpyod\&utm_medium=blog). The library for doing it is the Python package [**Fancy impute**](https://pypi.org/project/fancyimpute/).

### **REPRESENTATION LEARNING**

Hand-built and imputed features have an alternative: learning the representation itself. Representation Learning with Contrastive Predictive Coding is the [**paper**](https://arxiv.org/abs/1807.03748?utm_campaign=The%20Batch\&utm_source=hs_email\&utm_medium=email\&utm_content=83602348&_hsenc=p2ANqtz-8DcgUdDF--k3tWOmhM51lm28wHerZxlXKoGNU6hIu2P4Fj-RuEuciKtbWZZdWmvBg7KGeI44FWUrmpHdZIbAM-pUicgg&_hsmi=83602348) that proposes a universal unsupervised approach to extract useful representations from high-dimensional data, since unsupervised learning has not seen the adoption supervised learning has.

### **SIMILARITY**

Once rows are vectors, whether hand-built or learned, the next question is how alike two of them are. The same notes are in [Recommender Systems](/ai-product/recommender-systems) and [String Matching](/natural-language-processing-nlp/string-matching).

Cosine similarity is the default measure. Christian S. Perone's Cosine Similarity for Vector Space Models (Part III) is the continuation of his TF-IDF tutorial (Part I and Part II), and it is the [**Cosine similarity tutorial**](http://blog.christianperone.com/2013/09/machine-learning-cosine-similarity-for-vector-space-models-part-iii/). The Stack Exchange question [**Cosine vs dot product**](https://datascience.stackexchange.com/questions/744/cosine-similarity-versus-dot-product-as-distance-metrics) notes that cosine similarity is just the dot product scaled by the product of the magnitudes, and asks when each makes the better distance metric; the same Perone tutorial on its https address is [**Cosine vs dot product 2**](https://blog.christianperone.com/2013/09/machine-learning-cosine-similarity-for-vector-space-models-part-iii/). When speed matters, [**Fast cosine similarity**](https://stackoverflow.com/questions/51425300/python-fast-cosine-distance-with-cython) **implementation** is the Stack Overflow thread on speeding up scipy's cosine distance with numpy and cython.

Words and strings need their own measures. Edit distance similarity is one. [**Diff lib similarity and soundex**](https://datascience.stackexchange.com/questions/12575/similarity-between-two-words) is the question about a Python library to identify the similarity between two words or sentences, for comparing audio-to-text output against known words such as a person or company name. Selva Prabhakaran's Gensim tutorial, for the package billed as 'Topic Modeling for Humans' that also works with word vector models such as Word2Vec and FastText, covers [**Soft cosine and cosine**](https://www.machinelearningplus.com/nlp/gensim-tutorial/). Correlation is a similarity measure too: MachineLearningMastery's How to Calculate Correlation Between Variables in Python is the [**Pearson also used to detect similar vectors**](https://machinelearningmastery.com/how-to-use-correlation-to-understand-the-relationship-between-variables/) link.

### Distance

Similarity has a mirror image in distance, and some models need the distance form. The same notes are in [Dynamic Time Warping (DTW)](/predictive-ml/time-series-search#dynamic-time-warping-dtw).

MachineLearningMastery's 4 Distance Measures for Machine Learning is the [Mastery on distance formulas](https://machinelearningmastery.com/distance-measures-for-machine-learning/) link. It covers the role of distance measures, then Hamming Distance, Euclidean Distance, Manhattan Distance (Taxicab or City Block), and Minkowski Distance. The bridge back to the section above is that cosine distance = 1 - cosine similarity. Geo data needs one more measure: working with Geo data is fun once the data is clean and loaded to a dataframe or an array, and the real work starts when you have to find distances between two coordinates or cities and generate a distance matrix, which is where [Haversine](https://kanoki.org/2019/12/27/how-to-calculate-distance-in-python-and-pandas-using-scipy-spatial-and-distance-functions/) distance with Scipy spatial and pandas comes in.

#### Distance Tools

For geographic distance at scale, the tool is [GeoPandas](https://geopandas.org/en/stable/index.html), whose GeoPandas 1.1.4 documentation is the reference.

###

TF-IDF notes from this page are on the [TF-IDF](/natural-language-processing-nlp/tf-idf) page.


# Annotation & Disagreement

Labels are work product, and annotators disagree, so the page has to face myths, disagreement, and agreement metrics before it trusts a tool shelf. The notes move from myths through disagreement, inter-agreement, troubleshooting, crowdsourcing, and vision annotation, then the tools that run that loop. The same notes are in [Active Learning](/problem-framing/active-learning) and [Label Noise](/problem-framing/label-algorithms#label-noise).

## Myths

Most annotation projects start from beliefs about what a good label is, and the 7 myths of annotation, from "Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation" (its address is with the crowd-sourcing sources further down), argue that human annotation rests on an antiquated ideal of a single correct truth. The seven myths are:

1. Myth One: One Truth. Most data collection efforts assume that there is one correct interpretation for every input example.
2. Myth Two: Disagreement Is Bad. To increase the quality of annotation data, disagreement among the annotators should be avoided or reduced.
3. Myth Three: Detailed Guidelines Help. When specific cases continuously cause disagreement, more instructions are added to limit interpretations.
4. Myth Four: One Is Enough. Most annotated examples are evaluated by one person.
5. Myth Five: Experts Are Better. Human annotators with domain knowledge provide better annotated data.
6. Myth Six: All Examples Are Created Equal. The mathematics of using ground truth treats every example the same; either you match the correct result or not.
7. Myth Seven: Once Done, Forever Valid. Once human annotated data is collected for a task, it is used over and over with no update. New annotated data is not aligned with previous data.

## Disagreement

If disagreement is not simply bad, as the second myth claims, then it is something to measure rather than remove, and the rest of the page is how to measure it.

## Inter agreement

Measuring disagreement means choosing an agreement metric, and the choice depends on how many raters there are and what kind of labels they give. The same notes are in [Ground Truth](/natural-language-processing-nlp/sentiment-analysis#ground-truth).

The DKPro Agreement 2.0 slides from the Ubiquitous Knowledge Processing (UKP) Lab at Technische Universität Darmstadt are the place to start, marked \*\*\* in the original notes: [The best tutorial on agreements, cohen, david, kappa, krip etc.](https://dkpro.github.io/dkpro-statistics/inter-rater-agreement-tutorial.pdf)

The first metric is Cohens kappa, for two people, but you can use it to map a group by calculating agreement for each pair. Whether it belongs in classification evaluation is disputed. [Why cohens kappa should be avoided as a performance measure in classification](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0222916) shows that Cohen's Kappa and the Matthews Correlation Coefficient are correlated in most situations but differ in unbalanced ones, where Kappa can give a worse classifier a higher score. The other side is [Why it should be used as a measure of classification](https://thedatascientist.com/performance-measures-cohens-kappa-statistic/), which argues that Cohen's kappa statistic is a very good measure that can handle both multi-class and imbalanced class problems very well. For the intuition, [Kappa in plain english](https://stats.stackexchange.com/questions/82162/cohens-kappa-in-plain-english) is the question from a reader of a data mining book asking what kappa actually tells you about a classifier. [Multilabel using kappa](https://stackoverflow.com/questions/52272901/multi-label-annotator-agreement-with-cohen-kappa) is the case of two annotators labelling documents with multiple labels each and trying sklearn's cohen\_kappa\_score. [Kappa and the relation with accuracy](https://gis.stackexchange.com/questions/110188/how-are-kappa-and-overall-accuracy-related-with-respect-to-thematic-raster-data) is marked (redundant, % above chance, should not be used due to other reasons researched here).

The Kappa statistic varies from 0 to 1, where:

* 0 = agreement equivalent to chance.
* 0.1 – 0.20 = slight agreement.
* 0.21 – 0.40 = fair agreement.
* 0.41 – 0.60 = moderate agreement.
* 0.61 – 0.80 = substantial agreement.
* 0.81 – 0.99 = near perfect agreement
* 1 = perfect agreement.

With three people or more, Cohen's kappa gives way to Fleiss’ kappa, from 3 people and above. Kappa ranges from 0 to 1, where 0 is no agreement (or agreement that you would expect to find by chance) and 1 is perfect agreement. Fleiss’s Kappa is an extension of Cohen’s kappa for three raters or more. In addition, the assumption with Cohen’s kappa is that your raters are deliberately chosen and fixed. With Fleiss’ kappa, the assumption is that your raters were chosen at random from a larger population. Two other metrics cover the remaining cases: [Kendall’s Tau](https://www.statisticshowto.datasciencecentral.com/kendalls-tau/) is used when you have ranked data, like two people ordering 10 candidates from most preferred to least preferred, and Krippendorff’s alpha is useful when you have multiple raters and multiple possible ratings.

Krippendorfs alpha is the most general of them. The deepsense.ai post on multilevel classification, Cohen kappa and Krippendorff alpha, written while building a classifier that predicts cancer type from genetic signatures in The Genome Cancer Atlas, notes that it [Ignores missing data entirely](https://deepsense.ai/multilevel-classification-cohen-kappa-and-krippendorff-alpha/). It can handle various sample sizes, categories, and numbers of raters, and it applies to any [measurement level](https://www.statisticshowto.datasciencecentral.com/scales-of-measurement/), i.e. [nominal, ordinal, interval, ratio](https://www.statisticshowto.datasciencecentral.com/nominal-ordinal-interval-ratio/). Values range from 0 to 1, where 0 is perfect disagreement and 1 is perfect agreement. Krippendorff suggests: “\[I]t is customary to require α ≥ .800. Where tentative conclusions are still acceptable, α ≥ .667 is the lowest conceivable limit (2004, p. 241).” It is [Supposedly multi label](https://stackoverflow.com/questions/57256287/calculate-kappa-score-for-multi-label-image-classifcation), the question about computing kappa for multi-label image classification when sklearn's cohen\_kappa\_score raises "multilabel-indicator is not supported".

All of these metrics score the raters as a group; MACE - the new kid on the block - scores each one. It learns in an unsupervised fashion to a) identify which annotators are trustworthy and b) predict the correct underlying labels. We match performance of more complex state-of-the-art systems and perform well even under adversarial conditions. MACE does exactly that: it tries to find out which annotators are more trustworthy and upweighs their answers. The code is the Multi-Annotator Competence Estimation tool on [Git](https://github.com/dirkhovy/MACE).

When evaluating redundant annotations (like those from Amazon's MechanicalTurk), we usually want to

1. aggregate annotations to recover the most likely answer
2. find out which annotators are trustworthy
3. evaluate item and task difficulty

MACE solves all of these problems, by learning competence estimates for each annotators and computing the most likely answer based on those competences.

Calculating agreement then comes in three forms: compare against researcher-ground-truth, self-agreement, and inter-agreement. For inter-agreement, Amir Ziai's Inter-rater agreement Kappas on [Medium](https://medium.com/data-science/inter-rater-agreement-kappas-69cd8b91ff75) defines inter-rater reliability as the degree of agreement among raters, a score of how much homogeneity or consensus there is in the ratings. The [Kappa](https://stats.stackexchange.com/questions/82162/cohens-kappa-in-plain-english) plain-English thread above applies here too, as does [Multi annotator with kappa (which isnt), is this okay?](https://stackoverflow.com/questions/52272901/multi-label-annotator-agreement-with-cohen-kappa). For three raters or more, the Github gist to compute Fleiss' kappa using numpy is [1](https://gist.github.com/skylander86/65c442356377367e27e79ef1fed4adee), and [Fleiss Kappa Example](https://www.wikiwand.com/en/Fleiss'_kappa#/Worked_example) is the worked example. When ratings are scores rather than categories, [GWET AC1](https://stats.stackexchange.com/questions/235929/fleiss-kappa-alternative-for-ranking) is the answer to a question asking for a weighted alternative to Fleiss' kappa for scores from 1 to 6, and the [paper](https://s3.amazonaws.com/sitesusa/wp-content/uploads/sites/242/2014/05/J4_Xie_2013FCSM.pdf) behind it splits agreement measures into a classical descriptive approach and a modeling approach. A Website, krippensorf vs fleiss calculator, used to sit here and is kept at the end of the page.

## Troubling shooting agreement metrics

Metrics can still disagree with intuition, most often on imbalance data sets, i.e., why my agreement looks high while the metric is low. [Why is reliability so low when percentage of agreement is high?](https://www.researchgate.net/post/Why_is_reliability_so_low_when_percentage_of_agreement_is_high) is that exact question. [Interpretation of kappa values](https://medium.com/data-science/interpretation-of-kappa-values-2acd1ca7b18f) explains that the kappa statistic is frequently used to test interrater reliability, which represents the extent to which the collected data are correct representations of the variables measured. The notes on Interpreting agreement, Accuracy precision kappa used to sit here and are kept at the end of the page.

## Crowd Sourcing

Once agreement can be measured, the practical question is who should annotate: experts or a crowd.

#### Crowd Sourcing

Professor Kenneth Benoit's Alan Turing Institute talk, "Ground truth? The uses and abuses of human annotation in text analysis", is the video for this question.

{% embed url="<https://www.youtube.com/watch?v=ktZLuXPXPEI>" %}

The figures below are notes from that [Crowd Sourcing ](https://www.youtube.com/watch?v=ktZLuXPXPEI)talk.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-17dd52f7caf9e3d02634376773198bacc693b9f1%2Fgimg-3613f234815e.png?alt=media" alt=""><figcaption><p>Crowdsourcing annotation notes.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-9e094fb1396e587609346eddd69c289264976a9c%2Fgimg-94052c60284f.png?alt=media" alt=""><figcaption><p>Crowdsourcing annotation notes.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-e25418256d5c5430c8657a36c66b915da149ec0d%2Fgimg-10136540d579.png?alt=media" alt=""><figcaption><p>Crowdsourcing annotation notes.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-c800209865f3b125353efc5e9d3944d701d9a2ff%2Fgimg-b0c33763ac4e.png?alt=media" alt=""><figcaption><p>Crowdsourcing annotation notes.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-b38f179d05fd466defc0b627658acfd8a0a0fc1d%2Fgimg-6ebd6808923c.png?alt=media" alt=""><figcaption><p>Crowdsourcing annotation notes.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-a48b5839bfb6b7a08ba301226a9deda2fa773672%2Fgimg-bd667d6736a1.png?alt=media" alt=""><figcaption><p>Crowdsourcing annotation notes.</p></figcaption></figure>

The conclusions answer the fifth myth directly: experts are the same as a crowd, and the crowd costs a lot less $$$.

## Machine Vision annotation

Text is not the only thing a crowd labels; images need their own tooling. The same notes are in [Deep Neural Machine Vision](/deep-learning/deep-neural-machine-vision).

For vision, the tool to look at is [CVAT](https://venturebeat.com/2019/03/05/intel-open-sources-cvat-a-toolkit-for-data-labeling/).

The full addresses for the crowd-sourcing and weak-supervision sources used on this page are:

* Snorkel metal, from the weak-supervision tools below: <https://jdunnmon.github.io/metal_deem.pdf>
* Mturk alternatives, Lauren Bennett's list of the best MTurk alternatives, some of which pay more: <https://moneypantry.com/amazon-mechanical-turk-crowdsourcing-alternatives/>
* Jobby, now a site for job opportunities, career advice, and hiring tips: <https://www.jobboy.com/>
* Shorttask, which connects online job seekers with task providers: <http://www.shorttask.com/>
* Samasource, now Sama, a team of experts delivering data solutions: <https://www.samasource.org/team>
* The definite guide to Appen (Figure Eight) micro tasks: <https://www.earnonlineguys.com/figure-eight-tasks-guide/>
* 7 myths about annotation, the paper behind the Myths section: <https://www.aaai.org/ojs/index.php/aimagazine/article/viewFile/2564/2468>

## Tools

With the people and the metrics settled, the tools are what run the annotation loop day to day, starting with ways to need fewer human labels.

[Snorkel](https://www.snorkel.org/use-cases/) - using weak supervision to create less noisy labelled datasets. Its [Git](https://github.com/snorkel-team/snorkel) repo is the system for quickly generating training data with weak supervision; a Medium introduction to it used to sit here and is kept at the end of the page. Snorkel metal extends that to weak supervision for multi-task learning; the Conversation about it is kept at the end of the page, and the [git](https://github.com/HazyResearch/metal/blob/master/tutorials/Multitask.ipynb) notebook is the Snorkel MeTaL multi-task tutorial. The Snorkel team's answer on hierarchical labels explains where that work stands:

"Yes, the Snorkel project has included work before on hierarchical labeling scenarios. The main papers detailing our results include the DEEM workshop paper you referenced (<https://dl.acm.org/doi/abs/10.1145/3209889.3209898>) and the more complete paper presented at AAAI (<https://arxiv.org/abs/1810.02840>). Before the Snorkel and Snorkel MeTaL projects were merged in Snorkel v0.9, the Snorkel MeTaL project included an interface for explicitly specifying hierarchies between tasks which was utilized by the label model and could be used to automatically compile a multi-task end model as well (demo here: <https://github.com/HazyResearch/metal/blob/master/tutorials/Multitask.ipynb>). That interface is not currently available in Snorkel v0.9 (no fundamental blockers; just hasn't been ported over yet).

There are, however, still a number of ways to model such situations. One way is to treat each node in the hierarchy as a separate task and combine their probabilities post-hoc (e.g., P(credit-request) = P(billing) \* P(credit-request | billing)). Another is to treat them as separate tasks and use a multi-task end model to implicitly learn how the predictions of some tasks should affect the predictions of others (e.g., the end model we use in the AAAI paper). A third option is to create a single task with all the leaf categories and modify the output space of the LFs you were considering for the higher nodes (the deeper your hierarchy is or the larger the number of classes, the less apppealing this is w/r/t to approaches 1 and 2)."

When the labels do come from people, the budget comes first, and the Mechanical Turk Cost Calculator is the [mechanical turk calculator](https://morninj.github.io/mechanical-turk-cost-calculator/). The Mturk alternatives are Workforce / onespace, Jobby, Shorttask, Samasource, and Figure 8, with its pricing and the definite guide; their addresses are in the list above.

For doing the annotation itself, [Brat nlp annotation tool](http://brat.nlplab.org/) is a web-based annotation tool for textual annotation. [Prodigy by spacy](https://prodi.gy/) is a downloadable annotation tool for LLMs, NLP and computer vision tasks such as named entity recognition, text classification, object detection, image segmentation, and evaluation. Explosion's video TRAINING AN INSULTS CLASSIFIER with Prodigy in \~1 hour is the [seed-small sample, many sample tutorial on youtube by ines](https://www.youtube.com/watch?v=5di0KlKl0fE), and David Campion's "Text Classification: Be lazy, use Prodigy !" is [How to use prodigy, tutorial on medium plus notebook code inside](https://medium.com/@david.campion/text-classification-be-lazy-use-prodigy-b0f9d00e9495). [Doccano](https://github.com/chakki-works/doccano) - prodigy open source alternative butwith users management & statistics out of the box. The Medium announcement of [Lighttag - has some cool annotation metrics\tests](https://medium.com/@TalPerry/announcing-lighttag-the-easy-way-to-annotate-text-afb7493a49b8). Loopr.ai - An AI powered semi-automated and automated annotation process for high quality data.object detection, analytics, nlp, active learning; its address is kept at the end of the page.

Once annotations exist, the disagreement between them has to be read. Oliver Price's Medium post [Assessing annotator disagreement](https://medium.com/data-science/assessing-annotator-disagreements-in-python-to-build-a-robust-dataset-for-machine-learning-16c74b49f043) starts from Matthew Honnibal's 2018 PyData Berlin point that a solid annotation schema and attentive annotators are building blocks of a successful model. [A great python package for measuring disagreement on GH](https://github.com/o-P-o/disagree) is the package to visualise, evaluate, and manage annotated data. The Kenneth Benoit talk from the crowd-sourcing section is the reminder that [Reliability is key, and not just mechanical turk](https://www.youtube.com/watch?v=ktZLuXPXPEI), and the 7 myths about annotation above are the same warning in paper form.

Two annotation studies show the whole loop in practice. Multilingual Twitter Sentiment Classification: The Role of Human Annotators is [Annotating twitter sentiment using humans, 3 classes, 55% accuracy using SVMs.](http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0155036) It finds that model quality depends much more on the quality and size of training data than on the type of model; they talk about inter agreement etc. and their DS is [partially publicly available](https://www.clarin.si/repository/xmlui/handle/11356/1054) as Twitter sentiment for 15 European languages. [Exploiting disagreement ](https://web.eecs.umich.edu/~mihalcea/papers/chklovski.ranlp03.pdf)turns that disagreement into something to use rather than remove. [Vader annotation](https://web.archive.org/web/20160327132241/http://comp.social.gatech.edu/papers/icwsm14.vader.hutto.pdf) is the annotation protocol behind VADER, and its rules read as a checklist:

1. They must pass an english exam
2. They get control questions to establish their reliability
3. They get a few sentences over and over again to establish inter disagreement
4. Two or more people get a overlapping sentences to establish disagreement
5. 5 judges for each sentence (makes 4 useless)
6. They dont know each other
7. Simple rules to follow
8. Random selection of sentences
9. Even classes
10. No experts
11. Measuring reliability kappa/the other kappa.

For running all of that on one platform, [Label studio](https://labelstud.io/) is a multi-modal data labeling and annotation platform for agent traces, LLM evals, RLHF, computer vision, document AI, NLP, audio transcription, and more.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-33dc0a75f58a982fb207cb6fca8a53f6d3ff9cff%2Fgimg-31c51da0af26.png?alt=media" alt=""><figcaption><p>Label Studio.</p></figcaption></figure>

The tools also suggest ways to label less. Ideas:

1. Active learning for a group (or single) of annotators, we have to wait for all annotations to finish each big batch in order to retrain the model.
2. Annotate a small group, automatic labelling using knn
3. Find a nearest neighbor for out optimal set of keywords per “category,
4. For a group of keywords, find their knn neighbors in w2v-space, alternatively find k clusters in w2v space that has those keywords. For a new word/mean sentence vector in the ‘category’ find the minimal distance to the new cluster (either one of approaches) and this is new annotation.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Medium This address no longer opens: <https://towardsdatascience.com/introducing-snorkel-27e4b0e6ecff>
* Conversation This address no longer opens: <https://spectrum.chat/snorkel/help/hierarchical-labelling-example\\~aa4d8617-d287-43a6-865e-7c9034888363>
* Workforce / onespace This address no longer opens: <https://www.crowdsource.com/workforce/>
* pricing This address no longer opens: <https://siftery.com/crowdflower/pricing>
* Loopr This address no longer opens: <https://loopr.ai/products/labeling-platform>
* Assessing annotator disagreement This address no longer opens: <https://towardsdatascience.com/assessing-annotator-disagreements-in-python-to-build-a-robust-dataset-for-machine-learning-16c74b49f043>
* Exploiting disagreement This address no longer opens: <https://s3.amazonaws.com/academia.edu.documents/8026932/10.1.1.2.8084.pdf?AWSAccessKeyId=AKIAIWOWYYGZ2Y53UL3A\\&Expires=1534444363\\&Signature=3dHHw3EmAjPXFxwutVbtsZWEIzw%3D\\&response-content-disposition=inline%3B%20filename%3DExploiting\\_agreement\\_and\\_disagreement\\_of.pdf>
* Vader annotation This address no longer opens: <http://comp.social.gatech.edu/papers/icwsm14.vader.hutto.pdf>
* Medium This address no longer opens: <https://towardsdatascience.com/inter-rater-agreement-kappas-69cd8b91ff75>
* Website, krippensorf vs fleiss calculator This address no longer opens: <https://nlp-ml.io/jg/software/ira/>
* Interpretation of kappa values This address no longer opens: <https://towardsdatascience.com/interpretation-of-kappa-values-2acd1ca7b18f>
* Interpreting agreement This address no longer opens: <http://web2.cs.columbia.edu/\\~julia/courses/CS6998/Interrater\\_agreement.Kappa\\_statistic.pdf>
* MACE. This address no longer opens: <https://www.isi.edu/publications/licensed-sw/mace/>
* Medium. This address no longer opens: <https://medium.com/data-science/introducing-snorkel-27e4b0e6ecff>


# Next

Next is Analytics. Once features are scaled, transformed, imputed, and annotated, the transformed table has to be understood before a model family is chosen, because probability and dependence decide what a later score means.


# Analytics

Probability and dependence decide what a later score means. After this chapter the reader can read a distribution, a dependence, and an information measure, and can choose a feature for a reason.

The chapter starts with the shapes under the data. [Probability](/data/analytics-intro/probability) gives the basic probability pictures and the PDF they become, then kernel density estimation for when a histogram is sparse. A dataset always sits under some probability shape, so [Distribution](/data/analytics-intro/distribution) names that shape, looks at the Gaussian that so many tests assume, and then compares two distributions, including by distance. [Probability & Statistics](/data/analytics-intro/probability-and-statistics) steps back to the difference between the two fields: probability asks what happens next under a random process, and statistics asks what process would explain what already happened.

With the shapes named, the chapter turns to measures. A model that ranks classes or splits trees needs a measure of surprise and a measure of how wrong a predicted distribution is, and [Information Theory](/data/analytics-intro/information-theory) supplies entropy, information gain, cross-entropy, the divergence family, and softmax. [Data Analytics](/data/analytics-intro/data-analytics) is the practical path from raw tables to questions a team can answer without a model yet, laid out as a free course. The chapter ends with [Dependence and Selection](/data/analytics-intro/dependence-and-selection): features that move together waste capacity and features that do not predict the target waste the model, so that page measures dependence and then selects.

Analysis does not have to go through code alone. [Understanding PandasAI](https://cohenori.medium.com/understanding-pandasai-fc135b871e84) (May 2023) shows analysis through a generative interface: Pandas-AI is a Python library that adds generative AI to Pandas and makes data frames conversational, and the piece looks at how an LLM can be steered to answer data questions.


# Probability

Before a distribution or a density estimate can be read, the basic probability pictures have to be clear, and so does the PDF those pictures become. The page starts with crib figures, moves to the probability density function, and ends with kernel density estimation and the videos that teach it.

The same notes are in [Distribution](/data/analytics-intro/distribution) and [Probability & Statistics](/data/analytics-intro/probability-and-statistics).

## Probability crib figures

The first step is visual. These four crib sheets from [en.wikipedia.org](https://en.wikipedia.org/) put the basic probability ideas on one screen before any density function appears.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-9881280a3537a1481f04c1f38253f832b85b6645%2Fgimg-c0758b95a577.png?alt=media" alt=""><figcaption><p>Probability crib figure.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-46fd1dcaaa212bbbb0a4ce8913a6ce32e5eb9b70%2Fgimg-548a6386a5ac.png?alt=media" alt=""><figcaption><p>Probability crib figure.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-eb2a6cf1f7291e43765be2caac39035d0e869e54%2Fgimg-17d46a314d25.png?alt=media" alt=""><figcaption><p>Probability crib figure.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-cea248ab23a084fb9bbb9a91b52d7a1c6174a960%2Fgimg-e31bb8b7775a.png?alt=media" alt=""><figcaption><p>Probability crib figure.</p></figcaption></figure>

## PDF (probability density function)

Once the pictures are familiar, the next question is how to get a PDF out of real sample data in code. The [Tutorial in scipy](https://oneau.wordpress.com/2011/02/28/simple-statistics-with-scipy/) is the "Simple statistics with SciPy" post on Comfort at 1 AU, the starting point for doing this in SciPy. The [Array-based tutorial in python with PDF and KDE](http://firsttimeprogrammer.blogspot.co.il/2015/01/how-to-estimate-probability-density.html) shows how to estimate a probability density function from sample data with Python, on a blog of statistics and machine-learning experiments in Python, R, and Java. The [Summary of univariate distribution including pdf methods](https://www.johndcook.com/blog/distributions_scipy/) is a set of notes on probability distribution functions in Python using SciPy, useful as a reference once the first estimate works.

## Kernel Density Estimation

A PDF from a histogram breaks down when the histogram is sparse, and that is where KDE comes in. A tutorial that used to sit here actually explains why we should use KDE over a Histogram: it explains the cons of histograms and how KDE helps solve some issue that we usually encounter in ‘Sparse’ histograms where the distribution is hard to figure out. That address no longer opens and is kept at the end of the page.

For the code itself, there is supposedly a better [implementation](https://github.com/Daniel-B-Smith/KDE-for-SciPy), a repository of code meant to be integrated with SciPy's KDE module. Supposedly better KDE than SciPy is the author's note on it, not a benchmark.

How to use KDE? A [tutorial](http://pythonhosted.org/PyQt-Fit/KDE_tut.html) about kernel density and how to use it in python. Has several good graphs and shows use cases.

### Video tutorials about Kernel Density

Reading about KDE is slower than watching the bandwidth change, so the next step is video. Anders Munk-Nielsen's series starts with [KDE](https://www.youtube.com/watch?v=gPWsDh59zdo) itself, the Kernel Density Estimation lecture. The same series then widens to Non parametric [Kernel Regression Estimation](https://www.youtube.com/watch?v=ncF7ArjJFqM), then Non parametric [Sieve Estimation](https://www.youtube.com/watch?v=cqecz-DL-jI), and ends with [Semi- nonparametric estimation](https://www.youtube.com/watch?v=G1N53K530To), Robinson's double residual method.

The [Udacity Video Tutorial](https://www.youtube.com/watch?v=MEP35FcrQGs\&list=PLAwxTw4SYaPn-ttWkPiUL7NP3lLRdUniJ\&index=80) - pretty good - is Udacity's "Inspecting Distributions" lesson from Model Building and Validation.

After the videos, the written references compare and apply the methods. Jake VanderPlas's Kernel Density Estimation in Python post on Pythonic Perambulations is marked IMPORTANT: [Comparison and benchmarks of various KDE algo’s](https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/). Will Koehrsen's [Histograms and density plots](https://medium.com/data-science/histograms-and-density-plots-in-python-f6bda88f5ac0) starts from the simple histogram, which shows the location, spread, and shape of the data, and moves to density plots for the cases where a histogram fails. The [SK LEARN](http://scikit-learn.org/stable/modules/density.html#kernel-density-estimation) density-estimation guide frames the topic as walking the line between unsupervised learning, feature engineering, and data modeling, with mixture models among the most popular techniques. The same Udacity lesson on inspecting distributions is linked once more as [Gaussian KDE in scipy, version 2](https://www.youtube.com/watch?v=MEP35FcrQGs\&list=PLAwxTw4SYaPn-ttWkPiUL7NP3lLRdUniJ\&index=80).

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* This tutorial. This address no longer opens: <https://mglerner.github.io/posts/histograms-and-kernel-density-estimation-kde-2.html?p=28>


# Distribution

A dataset always sits under some probability shape, and the work starts by naming that shape before comparing two of them. The page moves from distribution types, through the Gaussian that so many tests assume, into comparing distributions and then distance methods between them. The same notes are in [Distribution Transformation](/data/processing-intro/distribution-transformation) and [Probability](/data/analytics-intro/probability).

## Types

Naming a shape starts with what a distribution is: (What are?) probabilities in a distribution always add up to 1. MachineLearningMastery's "A Gentle Introduction to Statistical Data Distributions" gives [More distribution explanations](https://machinelearningmastery.com/statistical-data-distributions/) for readers who want the definitions in prose before the pictures.

The fastest overview is a crib sheet. A very good explanation (Cloudera crib sheet) is the Cloudera Blog post on common probability distributions, the data scientist's crib sheet: <https://blog.cloudera.com/blog/2015/12/common-probability-distributions-the-data-scientists-crib-sheet/>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-cba0b6f500bf553dea74af2268a8a80d0e8b627e%2Fgimg-1b1a1d76129b.png?alt=media" alt=""><figcaption><p>Common probability distributions crib sheet.</p></figcaption></figure>

When the crib sheet is too terse, "Statistical Distributions" is [A very wordy explanation](http://people.stern.nyu.edu/adamodar/New_Home_Page/StatFile/statdistns.htm), and its second figure is below.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-449da774d12eb6d09ad77ca7512bb6b9971d2d7c%2Fgimg-3e477c0f7117.png?alt=media" alt=""><figcaption><p>Figure 2 from the wordy distribution explanation.</p></figcaption></figure>

One type gets its own note: Poisson and Poisson process, explained on Towards Data Science at <https://towardsdatascience.com/the-poisson-distribution-and-poisson-process-explained-4e2cb17d459>

## Gaussian / Normal Distribution

Among all those types, one shape is assumed far more often than the rest, and the folk rule says it plainly: “ if you collect data and it is not normal, “you need to collect more data”. The source of that line no longer opens and is kept at the end of the page.

For why the one-sample t-test assumes normality, the Stack Exchange question has [Beautiful graphs](https://stats.stackexchange.com/questions/116550/why-do-we-have-to-assume-normality-for-a-one-sample-t-test); it starts from the central limit theorem, under which the sampling distribution of the sample means is normal, so the standard error and a 95% confidence interval can be computed to accept or reject the null hypothesis.

The Quora answer on why we use the normal distribution sums it up in one sentence: of all continuous distributions over the real line with a given variance, the normal uniquely maximizes differential entropy. In plainer terms, [The normal distribution is popular for two reasons:](https://www.quora.com/Why-do-we-use-the-normal-distribution-The-normal-is-an-approximation-Why-dont-we-use-a-simpler-distribution-with-simpler-numbers-to-memorize-If-it-is-an-approximation-does-it-have-to-be-so-specific)

1. It is the most common distribution in nature (as distributions go)
2. An enormous number of statistical relationships become clear and tractable if one assumes the normal.

Sure, nothing in real life exactly matches the Normal. But it is uncanny how many things come close.

This is partly due to the Central Limit Theorem, which says that if you average enough unrelated things, you eventually get the Normal.

The Normal distribution in statistics is a special world in which the math is straightforward and all the parts fit together in a way that is easy to understand and interpret. It may not exactly match the real world, but it is close enough that this one simplifying assumption allows you to predict lots of things, and the predictions are often pretty reasonable. It is statistically convenient, and it is represented by basic statistics: the average, and the variance (or standard deviation) - the average of what's left when you take away the average, but to the power of 2.

That convenience has a price in testing. In a statistical test, you need the data to be normal to guarantee that your p-values are accurate with your given sample size.

If the data are not normal, your sample size may or may not be adequate, and it may be difficult for you to know which is true.

## Comparing distributions

Once the shape of one sample is known, the next question is whether two samples share it. The classic test is Kolmogorov–Smirnov, and the Wikipedia entry carries the author's warning in its link: [Kolmogorov Smirnov not good for categoricals.](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test) The general question, how to check two unknown distributions for equality and which statistical tests exist for it, is the Math Stack Exchange thread on [Comparing two](https://math.stackexchange.com/questions/159940/comparing-distribution-of-two-data-sets). For the intuition behind describing and comparing distributions, there is [Khan academy](https://www.khanacademy.org/math/ap-statistics/quantitative-data-ap/describing-comparing-distributions/v/comparing-distributions). To compare them [Visually](https://www.stat.auckland.ac.nz/~ihaka/787/lectures-distrib.pdf), the lecture notes linked there are the reference.

The earlier Gaussian section warned about data that are not normal, and the Quora question on which test quantifies similarity between two distributions covers exactly the case [When they are not normal](https://www.quora.com/Which-statistical-test-to-use-to-quantify-the-similarity-between-two-distributions-when-they-are-not-normal). In machine learning the two samples are often train and test, and Shikhar Gupta's "How (dis)similar are my train and test data?" is the [Using train / test trick](https://medium.com/data-science/how-dis-similar-are-my-train-and-test-data-56af3923de9b): comparing one mix of apples and oranges with another mix whose distribution is different. Finally, when the goal is to name the shape of real data rather than compare two samples, there is [Code for Identifying distribution type and params, based on best fit.](https://stackoverflow.com/questions/37487830/how-to-find-probability-distribution-and-parameters-for-real-data-python-3), a Stack Overflow question on finding the distribution type and parameters that a skewed, all-positive target most closely resembles.

## Comparing distributions (distance methods)

A test says whether two distributions differ; a distance says by how much, which is what drift monitoring needs. Categorical data can be transformed to a histogram i.e., #class / total and then measured for distance between two histograms’, e.g., train and production. Using earth mover distance [python](https://jeremykun.com/2018/03/05/earthmover-distance/) [git wrapper to c](https://github.com/pdinges/python-emd), linear programming, so its slow. The python link is the Earthmover Distance post, which starts from computing distance between points with uncertain locations; the git wrapper to c is a Python wrapper for Yossi Rubner's implementation of the earth mover's distance (EMD).

The same notes are in [Drift](/ai-engineering/mlops-monitoring-and-alerts#drift).

For the idea itself, [Earth movers](https://medium.com/data-science/earth-movers-distance-68fff0363ef2) is the Medium explanation of earth mover's distance, and the [EMD paper](http://infolab.stanford.edu/pub/cstr/reports/cs/tr/99/1620/CS-TR-99-1620.ch4.pdf) is the paper behind it. Also check KL DIVERGENCE in the information theory section.

The same notes are in [Cross entropy, relative ent, KL-D, JS-D, soft max](/data/analytics-intro/information-theory#cross-entropy-relative-ent-kl-d-js-d-soft-max).

Distances between distributions can also drive learning, not only monitoring. [Bengio](https://arxiv.org/abs/1901.10912) et al, transfer objective for learning to disentangle casual mechanisms - We propose to meta-learn causal structures based on how fast a learner adapts to new distributions arising from sparse distributional changes

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* “ if you collect data and it is not normal, “you need to collect more data”. This address no longer opens: <https://www.isixsigma.com/topic/normal-distributions-why-does-it-matter/>
* Using train / test trick. This address no longer opens: <https://towardsdatascience.com/how-dis-similar-are-my-train-and-test-data-56af3923de9b>
* Earth movers. This address no longer opens: <https://towardsdatascience.com/earth-movers-distance-68fff0363ef2>


# Probability & Statistics

Probability asks what happens next under a random process; statistics asks what process would explain what already happened. The page starts from that difference, then walks introductions to probability and statistics, more statistics, sampling and resampling, wiki topics, and recommended courses. The same notes are in [Probability](/data/analytics-intro/probability).

The best single starting point is a [Coursera course](https://www.youtube.com/watch?v=WkOinijQmPU\&list=PLpl-gQkQivXiBmGyzLrUjzsblmQsLtkzJ\&index=1) on probabilities - for data science, actually quite good in explaining a lot of the basic tools,prob, conditional, distributions, sampling, CI, hypothesis, etc. A great resource for proba/bayes/b-networks/etc (adam bali) used to sit beside it; that address no longer opens and is kept at the end of the page.

### Difference between

Before any tool, it helps to see that the two fields invert the same uncertainty problem. The Stack Exchange question [Difference between](https://stats.stackexchange.com/questions/665/whats-the-difference-between-probability-and-statistics) asks what separates probability from statistics and why they are studied together. I.e, Probability deals with predicting the likelihood of future events, while statistics involves the analysis of the frequency of past events. The problems considered by probability and statistics are inverse to each other.

In probability theory we consider some underlying process which has some randomness or uncertainty modeled by random variables, and we figure out what happens.

\=> Underlying process + randomness and random variables -> what happens next?

In statistics we observe something that has happened, and try to figure out what underlying process would explain those observations.

\=> observe what happened -> what is the underlying process?

Finally, probability theory is mainly concerned with the deductive part, statistics with the inductive part of modeling processes with uncertainty.

### Introduction to Probability

The deductive side comes first, and Math is Fun walks it one small page at a time, starting from the observation that life is full of random events. [Types of events](https://www.mathsisfun.com/data/probability-events-types.html) names them, [Independent events](https://www.mathsisfun.com/data/probability-events-independent.html) covers events that do not affect each other, and [Conditional proba](https://www.mathsisfun.com/data/probability-events-conditional.html) is how to handle dependent events. Calculating probabilities can be hard, sometimes we add them, sometimes we multiply them, and often it is hard to figure out what to do, which is why [Proba tree diagrams](https://www.mathsisfun.com/data/probability-tree-diagrams.html) come next. [Mutually exclusive events](https://www.mathsisfun.com/data/probability-events-mutually-exclusive.html) are the ones that cannot happen at the same time, like turning left and turning right, or heads and tails.

Counting comes before most of those calculations. In English we use the word combination loosely, without thinking if the order of things is important, and [Combination and permutations](https://www.mathsisfun.com/combinatorics/combinations-permutations.html) makes that distinction exact. [Bayes](https://www.mathsisfun.com/data/bayes-theorem.html) is the Math is Fun page on Bayes' theorem. [Least squares regresssion](https://www.mathsisfun.com/data/least-squares-regression.html) It works by making the total of the square of the errors as small as possible (that is why it is called "least squares".

The sequence ends with random variables. A Random Variable is a set of possible values from a random experiment, such as heads and tails given the values 0 and 1. [Random variables](https://www.mathsisfun.com/data/random-variables.html) introduces them, [Continuous random variables](https://www.mathsisfun.com/data/random-variables-continuous.html) extends them past discrete outcomes, and [Random vars mean, std, variance](https://www.mathsisfun.com/data/random-variables-mean-variance.html) gives their mean, variance, and standard deviation.

### Introduction to statistics

With random variables in hand, the inductive side starts from data itself. Data is a collection of facts and numbers, and the Math is Fun [Table of](https://www.mathsisfun.com/data/index.html#stats) contents for using and handling data is the index for everything below.

The first statistics are centers. The Median is the middle of a sorted list of numbers ([Median](https://www.mathsisfun.com/median.html)), and the mode is the number which appears most often ([Mode](https://www.mathsisfun.com/mode.html)). The mean has several forms: the [Weighted mean](https://www.mathsisfun.com/data/weighted-mean.html); the [Geometric mean](https://www.mathsisfun.com/numbers/geometric-mean.html), a special type of average where we multiply the numbers together and then take a square root (for two numbers), cube root, and so on; and the [Harmonic mean](https://www.mathsisfun.com/numbers/harmonic-mean.html), the reciprocal of the average of the reciprocals.

Centers then need spread and position. Percentile is the value below which a percentage of data falls ([Percentiles](https://www.mathsisfun.com/data/percentiles.html)). [Mean deviation](https://www.mathsisfun.com/data/mean-deviation.html) is how far, on average, all values are from the middle. When two sets of data are strongly linked together we say they have a High Correlation ([Correlation](https://www.mathsisfun.com/data/correlation.html)). Deviation means how far from the normal, and the [Standard deviation](https://www.mathsisfun.com/data/standard-deviation.html) is a measure of how spread out numbers are; the [formula](https://www.mathsisfun.com/data/standard-deviation-formulas.html) page gives the calculation, deviation meaning how far from the average.

Spread leads to shape. Data can be distributed (spread out) in different ways, and in many cases it tends to sit around a central value, which is the [Standard normal distribution](https://www.mathsisfun.com/data/standard-normal-distribution.html). Data can be skewed, meaning it tends to have a long tail on one side or the other, and [Skewness of distribution](https://www.mathsisfun.com/data/skewness.html) explains why it is called negative skew.

Shape then supports inference. A confidence interval is a range of values we are fairly sure our true value lies in: [Confidence intervals (using std)](https://www.mathsisfun.com/data/confidence-interval.html). Accuracy is how close a measured value is to the true value and precision is how close the measurements are to each other, the [Accuracy vs precision (accurate vs hitting closely or density)](https://www.mathsisfun.com/accuracy-precision.html) page. [Probability](https://www.mathsisfun.com/data/probability.html) is how likely something is to happen, and the complement of an event is all outcomes that are NOT the event ([Probability complement](https://www.mathsisfun.com/data/probability-complement.html)). Is that a true difference, or just a fluke? The [Chi-square test, p\_value, independent, dependent, significance](https://www.mathsisfun.com/data/chi-square-test.html) page answers that by telling a real relationship from random chance.

Two terms get confused often enough to need their own links. [Variation vs variance](https://stats.stackexchange.com/questions/88348/is-variation-the-same-as-variance) - a private case - is the Stack Exchange question on whether variation and variance are the same term. [Std vs variance](https://www.investopedia.com/ask/answers/021215/what-difference-between-standard-deviation-and-variance.asp) - std is in the same metric as the mean, is the root of variance., allows outliers to influence, will not result in samples cancelling each other without the square root in the formula.

### More on Statistics

After the introductions, the same concepts come back in plain English as a Data Science Central series, in three parts. The first is [25 concepts](https://www.datasciencecentral.com/profiles/blogs/25-statistical-concepts-explained-in-simple-english-part-2), then (part 2) [29 more concepts](https://www.datasciencecentral.com/profiles/blogs/29-statistical-concepts-explained-in-simple-english-part-1), then (part1) & [part 3](https://www.datasciencecentral.com/profiles/blogs/29-statistical-concepts-explained-in-simple-english-part-2?fbclid=IwAR0VQFeBaJsm3ouEf7sV5WAupE1cI3PXhzWe9-lUYkZ_XCCF72_3r8w5hrI).

### STATISTICAL SAMPLING AND RESAMPLING

Every statistic above is computed on a sample, so the next question is how the sample was drawn and how wrong it can be.

The same notes are in [TRAIN / TEST / CROSS VALIDATION](/data/datasets-intro/datasets#train--test--cross-validation).

MachineLearningMastery's "A Gentle Introduction to Statistical Sampling and Resampling" answers [What is? Method for sampling/resampling, and sampling errors explained.](https://machinelearningmastery.com/statistical-sampling-and-resampling/)

Another great course on probability, distribution types, conditional, joint, chain, etc. <http://legacydirs.umiacs.umd.edu/~jbg/teaching/INST_414/>

### Wiki

Sampling from more than one variable at once needs the joint vocabulary, and the Wikipedia entries give the definitions. [Marginal probability](https://en.wikipedia.org/wiki/Marginal_distribution) is the marginal distribution entry, [Joint probability](https://en.wikipedia.org/wiki/Joint_probability_distribution) is the joint probability distribution entry, and [Conditional probability](https://en.wikipedia.org/wiki/Probability) points at the main Probability entry. The chain rule has two meanings worth separating: the probability \[Chain rule]\(<https://en.wikipedia.org/wiki/Chain\\_rule\\_(probability)> on Wikipedia, and derivatives using the chain rule, on [khan](https://www.khanacademy.org/math/ap-calculus-ab/ab-differentiation-2-new/ab-3-1a/v/chain-rule-introduction).

### Recommended Courses

The shelf closes with courses and one friendly article. Another great course on probability, distribution types, conditional, joint, chain, etc. is the course listed under sampling above. [Kahn](https://www.khanacademy.org/math/precalculus/prob-comb) is the probability course on the same site as the khan chain-rule video. Brandon Rohrer's "How Bayes Theorem works" is [A really good intro](https://www.youtube.com/watch?v=5NMxiOGL39M). [What are confidence intervals?](https://medium.com/data-science/a-very-friendly-introduction-to-confidence-intervals-9add126e714) is a very friendly introduction that discusses only the general idea, without too much fancy statistics terms, and with python.

(another angle) [The main difference between probability and statistics has to do with knowledge](https://www.thoughtco.com/probability-vs-statistics-3126368): what are the known facts? Inherent in both probability and statistics is a [population](https://www.thoughtco.com/what-is-a-population-in-statistics-3126308), every individual we are interested in studying, and a sample, consisting of the individuals that are selected from the population. In probability, we would start with us knowing everything about the composition of a population, and then would ask, “What is the likelihood that a selection, or sample, from the population, has certain characteristics?” In statistics, we have no knowledge about the types of socks in the drawer; we infer properties about the population on the basis of a random sample.

Some [calculations](https://www.mathsisfun.com/data/probability.html) to get you into probability:

* Finding out the probability of an event
* Of two consecutive events (multiplication)
* Of several events (sum)
* Etc..

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* A great resource for proba/bayes/b-networks/etc (adam bali). This address no longer opens: <https://metacademy.org/graphs/concepts/bayesian\\_networks#focus=i9mo2e09\\&mode=learn>
* What are confidence intervals? This address no longer opens: <https://towardsdatascience.com/a-very-friendly-introduction-to-confidence-intervals-9add126e714>


# Information Theory

A model that ranks classes or splits trees needs a measure of surprise and a measure of how wrong a predicted distribution is. The page starts with entropy and information gain, moves through the cross-entropy and divergence family and softmax, and then turns to tools, tutorials, time-series entropy, and complement objective training. The same notes are in [Collocation](/natural-language-processing-nlp/foundation-nlp#collocation) and [MUTUAL INFORMATION COEFFICIENT](/data/analytics-intro/dependence-and-selection#mutual-information-coefficient).

## Entropy / Information Gain

Surprise comes first, because every later measure on the page is built from it.

The same notes are in [Active Learning](/problem-framing/active-learning).

Shannon entropy in python is, basically, entropy(value counts): count how often each value appears, turn the counts into probabilities, and sum p log p. MachineLearningMastery's "A Gentle Introduction to Information Entropy" is the [Mastery on plogp entropy function](https://machinelearningmastery.com/what-is-information-entropy/) walkthrough, and the [Entropy functions](https://gist.github.com/jaradc/eeddf20932c0347928d0da5a09298147) gist shows four different ways to calculate entropy in Python.

## Cross entropy, relative ent, KL-D, JS-D, soft max

Entropy measures one distribution; training needs a measure between two, the true one and the predicted one.

The same notes are in [Comparing distributions (distance methods)](/data/analytics-intro/distribution#comparing-distributions-distance-methods), [LOSS](/deep-learning/deep-neural-nets#loss), and [LOSS IN KERAS](/deep-learning/deep-neural-frameworks#loss-in-keras).

Kullback–Leibler divergence is a very useful way to measure the difference between two probability distributions, and Count Bayesie's "Kullback-Leibler Divergence Explained" is [A really good explanation on all of them](https://www.countbayesie.com/blog/2017/5/9/kullback-leibler-divergence-explained). From the loss side, Raúl Gómez Bruballa's post on categorical and binary cross-entropy, softmax loss, logistic loss, focal loss, and all those confusing names is [Another good one on all of them](https://gombru.github.io/2018/05/23/cross_entropy_loss/). MachineLearningMastery's "A Gentle Introduction to Cross-Entropy for Machine Learning" is the [mastery on a gentle intro to CE](https://machinelearningmastery.com/cross-entropy-for-machine-learning/), and the [Mastery on entropy](https://machinelearningmastery.com/divergence-between-probability-distributions/) follow-up covers divergence between distributions: kullback leibler divergence (asymmetry), jensen-shannon divergence (symmetry) (has code).

The same family is tied together in [Entropy, mutual information and KL Divergence by AurelienGeron](https://www.techleer.com/articles/496-a-short-introduction-to-entropy-cross-entropy-and-kl-divergence-aurelien-geron/). Gensim also has notes on divergence metrics such as KL jaccard etc, pros and cons, lda is a mess on small data. For computing it by hand, the Stack Exchange question on calculating KL divergence in Python, where every pair of lists returned the same value, is the [Advise on KLD](https://datascience.stackexchange.com/questions/9262/calculating-kl-divergence-in-python)ivergence thread. Neural machine translation using pytorch and CE is where the same loss shows up in a full model.

## Softmax

Cross-entropy needs a predicted distribution to compare against, and softmax is what turns a network's scores into one.

The same notes are in [ACTIVATION FUNCTIONS](/deep-learning/deep-neural-nets#activation-functions) and [Temperature](/responsible-ai/calibration#temperature).

[Understanding softmax](https://medium.com/data-science-bootcamp/understand-the-softmax-function-in-minutes-f3a59641e86d) is Uniqtech Learning's short explanation of the activation function that turns logits into probabilities that sum to one. LJ V. MIRANDA's notebook, [Softmax and negative likelihood (NLL)](https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/), explains the softmax function, its relationship with the negative log-likelihood, and its derivative during backpropagation. The naming question is settled in [Softmax vs cross entropy](https://www.quora.com/Is-the-softmax-loss-the-same-as-the-cross-entropy-loss) - Softmax loss and cross-entropy loss terms are used interchangeably in industry. Technically, there is no term as such Softmax loss. people use the term "softmax loss" when referring to "cross-entropy loss". The softmax classifier is a linear classifier that uses the cross-entropy loss function. In other words, the gradient of the above function tells a softmax classifier how exactly to update its weights using some optimization like [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent).

The softmax() part simply normalises your network predictions so that they can be interpreted as probabilities. Once your network is predicting a probability distribution over labels for each input, the log loss is equivalent to the cross entropy between the true label distribution and the network predictions. As the name suggests, softmax function is a “soft” version of max function. Instead of selecting one maximum value, it breaks the whole (1) with maximal element getting the largest portion of the distribution, but other smaller elements getting some of it as well.

This property of softmax function that it outputs a probability distribution makes it suitable for probabilistic interpretation in classification tasks.

Cross entropy indicates the distance between what the model believes the output distribution should be, and what the original distribution is. Cross entropy measure is a widely used alternative of squared error. It is used when node activations can be understood as representing the probability that each hypothesis might be true, i.e. when the output is a probability distribution. Thus it is used as a loss function in neural networks which have softmax activations in the output layer.

## Tools

Once the measures are clear, computing them should not mean writing them from scratch. [EntroPy](https://raphaelvallat.com/entropy/build/html/index.html) is the entropy 0.1.3 documentation, and [AntroPy](https://raphaelvallat.com/antropy/) has its own 0.2.2 documentation; the AntroPy \[[Git](https://github.com/raphaelvallat/antropy) repo describes it as entropy and complexity of (EEG) time-series in Python. [PyInform](https://github.com/ELIFE-ASU/PyInform) \[[Docs](https://elife-asu.github.io/PyInform/index.html)]- PyInform is a python library of information-theoretic measures for time series data. PyInform is backed by the [Inform](https://github.com/elife-asu/inform) C library, a cross platform library for information analysis of dynamical systems. [PyEntropy](https://github.com/nikdon/pyEntropy) is one more package, entropy for Python.

## Tutorials

The libraries compute the numbers; the tutorials show why a decision tree wants them.

The same notes are in [CART TREES](/predictive-ml/decision-trees#cart-trees).

The scikit-learn decision tree learning tutorial on entropy, Gini, and information gain is the [Great tutorial on all of these topics](https://www.bogotobogo.com/python/scikit-learn/scikt_machine_learning_Decision_Tree_Learning_Informatioin_Gain_IG_Impurity_Entropy_Gini_Classification_Error.php).

[Entropy](https://www.techleer.com/articles/496-a-short-introduction-to-entropy-cross-entropy-and-kl-divergence-aurelien-geron/) - lack of order or lack of predictability ([excellent slide lecture by Aurelien Geron](https://www.youtube.com/watch?time_continue=3\&v=ErfnhcEV1O8)). The figures below come from that short introduction to entropy, cross-entropy, and KL-divergence.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-232e21f981595fcafcd2c813244af5c086364f40%2Fgimg-102a11303beb.png?alt=media" alt=""><figcaption><p>Entropy, cross-entropy, and KL divergence (Aurelien Geron).</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-f0657761639809597d9a43ed9622b36946a62478%2Fgimg-785fe96bfd79.png?alt=media" alt=""><figcaption><p>Entropy, cross-entropy, and KL divergence continued.</p></figcaption></figure>

Cross entropy will be equal to entropy if the probability distributions of p (true) and q(predicted) are the same. However, if cross entropy is bigger (known as relative\_entropy or kullback leibler divergence)

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-2f8c0fdc2c892966637764f1cabba50d12716723%2Fgimg-f763f8ab0174.png?alt=media" alt=""><figcaption><p>Relative entropy / Kullback–Leibler divergence when p and q differ.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-0d3d1c1a312aabf2e276c20e93ab8a435b854160%2Fgimg-2a9b4a8685c1.png?alt=media" alt=""><figcaption><p>Cross-entropy versus entropy illustration.</p></figcaption></figure>

In this example we want the cross entropy loss to be zero, i.e., when we have a one hot vector and a predicted vector which are identical, i.e., 100% in the same class for predicted and true, we get 0. In all other cases we get some number that gets larger if the predicted class probability is lower than zero as seen here:

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-6ccc0333eb24f1106d18a2df13303ee6e775a2f2%2Fgimg-7f0ca579b802.png?alt=media" alt=""><figcaption><p>Cross-entropy loss with one-hot true and predicted class probabilities.</p></figcaption></figure>

Formula for 2 classes:

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-fd85d16e4b23b47416346d7e077c4fcc98573c4c%2Fgimg-2aef43320fd6.png?alt=media" alt=""><figcaption><p>Cross-entropy formula for 2 classes.</p></figcaption></figure>

NOTE: Entropy can be generalized as a formula for N > 2 classes:

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-4ef06aec65f1667855c9532f1201d49d95d1e1ea%2Fgimg-f9ecbbfbc2b3.png?alt=media" alt=""><figcaption><p>Entropy generalized as a formula for N > 2 classes.</p></figcaption></figure>

The same loss logic now carries over to trees. The notes below follow an awesome pdf tutorial whose address no longer opens and is kept at the end of the page. (We want to grow a simple tree): a good attribute prefers attributes that split the data so that each successor node is as pure as possible, i.e., the distribution of examples in each node is so that it mostly contains examples of a single class. In other words: We want a measure that prefers attributes that have a high degree of „order“. Maximum order: All examples are of the same class. Minimum order: All classes are equally likely → Entropy is a measure for (un-)orderedness. Another interpretation: Entropy is the amount of information that is contained, and all examples of the same class → no information.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-bef661a583c5feddde539496f4a00697914500a7%2Fgimg-c3fc62bc4be7.png?alt=media" alt=""><figcaption><p>Entropy as unorderedness in the class distribution of S.</p></figcaption></figure>

Entropy is the amount of unorderedness in the class distribution of S. In the IMAGE above it has its maximal value when the equal class distribution holds, and its minimal value when only one class is in S.

That is the entropy of one node; a split has one node per category. So basically if we have the outlook attribute and it has 3 categories, we calculate the entropy for E(feature=category) for all 3.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-a5277b5e39cddb744c233dbe8f5113ef0f371715%2Fgimg-fbedcd82edef.png?alt=media" alt=""><figcaption><p>Entropy for each category of an attribute such as Outlook.</p></figcaption></figure>

INFORMATION: The I(S,A) formula below.

What we actually want is the average entropy of the entire split, that corresponds to an entire attribute, i.e., OUTLOOK (sunny & overcast & rainy)

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-cd50a27ab68c335fd284ebb371897de0fbf7ab9f%2Fgimg-15b45e8fd2fd.png?alt=media" alt=""><figcaption><p>The I(S,A) average entropy of an entire attribute split.</p></figcaption></figure>

Information Gain: is actually what we gain by subtracting information from the entropy.

In other words we find the attributes that maximizes that difference, in other other words, the attribute that reduces the unorderness / lack of order / lack of predictability.

The BIGGER GAIN is selected.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-cfd65cadf68ba8131a350b6ed6b7e96ccec4179e%2Fgimg-9eeba04d914d.png?alt=media" alt=""><figcaption><p>Information Gain as entropy minus information.</p></figcaption></figure>

There are some properties to Entropy that influence INFO GAIN (?):

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-20e4601bd5190a4cc094a42dbf743eb865fcb91e%2Fgimg-16f412550095.png?alt=media" alt=""><figcaption><p>Properties of Entropy that influence Information Gain.</p></figcaption></figure>

There are some disadvantages with INFO GAIN, done use it when an attribute has many number values, such as “day” (date wise) 05/07, 06/07, 07/07..31/07 etc.

Information gain is biased towards choosing attributes with a large number of values, and that bias causes overfitting and fragmentation.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-af8a5f0ad2d7f30ec9b8989bb2666ae58981fd3c%2Fgimg-b7008514d45b.png?alt=media" alt=""><figcaption><p>Information Gain bias toward attributes with many values.</p></figcaption></figure>

We measure Intrinsic information of an attribute, i.e., Attributes with higher intrinsic information are less useful.

We define Gain Ratio as info-gain with less bias toward multi value attributes, ie., “days”

NOTE: Day attribute would still win with the Gain Ratio, Nevertheless: Gain ratio is more reliable than Information Gain

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-4b86455d2c66a07863fd12651fd732a0d96b0f30%2Fgimg-03fde770bac0.png?alt=media" alt=""><figcaption><p>Gain Ratio as information gain with less multi-value bias.</p></figcaption></figure>

Therefore, we define the alternative, which is the GINI INDEX. It measures impurity, we define the average Gini, and the Gini Gain.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-3c1533969311a9bf0f60deca5bd1963940d96df6%2Fgimg-0c4b7812badf.png?alt=media" alt=""><figcaption><p>Gini index, average Gini, and Gini Gain.</p></figcaption></figure>

FINALLY, further reading about decision trees and examples of INFOGAIN and GINI here used to point at the same pdf tutorial, which is kept at the end of the page.

Beyond trees, mutual information itself is hard to estimate, and [Variational bounds on mutual information](https://arxiv.org/abs/1905.06922v1) is the arXiv paper "On Variational Bounds of Mutual Information".

## Time series entropy

The tree measures treat each row on its own; a time series asks how predictable the sequence is.

The same notes are in [Timeseries](/predictive-ml/forecasting).

[entroPY](https://raphaelvallat.com/entropy/build/html/index.html) - EntroPy is a Python 3 package providing several time-efficient algorithms for computing the complexity of one-dimensional time-series. It can be used for example to extract features from EEG signals. The [Approximate entropy paper](https://journals.physiology.org/doi/pdf/10.1152/ajpheart.2000.278.6.H2039) is behind one of its measures, approximate entropy. The calls below compute each of them on a series x:

```python
print(perm_entropy(x, order=3, normalize=True)) # Permutation entropy
print(spectral_entropy(x, 100, method='welch', normalize=True)) # Spectral entropy
print(svd_entropy(x, order=3, delay=1, normalize=True)) # Singular value decomposition entropy
print(app_entropy(x, order=2, metric='chebyshev')) # Approximate entropy
print(sample_entropy(x, order=2, metric='chebyshev')) # Sample entropy
print(lziv_complexity('01111000011001', normalize=True)) # Lempel-Ziv complexity
```

For measures beyond those, the [PyInform](https://elife-asu.github.io/PyInform/index.html) 0.2.0 documentation lists the information-theoretic measures shown in the two figures below.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-76ffd49cdd255144e0c9518e4af7f754bc0c3d3e%2Fgimg-15c13866e888.png?alt=media" alt=""><figcaption><p>PyInform information-theoretic measures.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-812e064fb6fd26654a0fab0c4d9d6679b08cff6e%2Fgimg-13adb68b8f41.png?alt=media" alt=""><figcaption><p>PyInform measures continued.</p></figcaption></figure>

## Complement Objective Training

Cross-entropy, from earlier on the page, only pushes up the correct class; complement objective training also uses the wrong ones. Complement Objective Training is a simple way to use incorrect-class probabilities and get more from labeled data in PyTorch Lightning. Article by [LightTag](https://www.lighttag.io/blog/complement-objective-training-with-pytorch-lightning/), and the method's [paper](https://arxiv.org/pdf/1903.01182.pdf) is on arXiv.

COT is a technique to effectively provide explicit negative feedback to our model. The technique gives us non-zero gradients with respect to incorrect classes, which are used to update the model's parameters.

COT doesn't replace cross-entropy. It's used as a second training step as follows: We run cross-entropy, and then we do a COT step. We minimize the cross-entropy between our target distribution. That's equivalent to maximizing the likelihood of the correct class. During the COT step, we maximize the entropy of the complement distribution. We pretend that the correct class isn't an option and make the remaining classes equally likely.

But, since the true class is an option, and we're training for it explicitly, maximizing the true classes probability and pushing the remaining classes to be equally likely is actually pushing their probabilities to 0 explicitly, which provides explicit gradients to propagate through our model.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Entropy. This address no longer opens: <https://www.techleer.com/articles/496-a-short-introduction-to-entropy-cross-entropy-and-kl-divergence-aurelien-geron/>
* awesome pdf tutorial. This address no longer opens: <http://www.ke.tu-darmstadt.de/lehre/archiv/ws0809/mldm/dt.pdf>
* FINALLY, further reading about decision trees and examples of INFOGAIN and GINI here.. This address no longer opens: <http://www.ke.tu-darmstadt.de/lehre/archiv/ws0809/mldm/dt.pdf>


# Data Analytics

Analytics is the path from raw tables to questions a team can answer without a model yet. This page is that path as a free data analytics course, walked in videos from spreadsheets and dashboards through SQL to Python and Pandas.

## Course

The course starts with what the job is before any tool. CareerFoundry's [What Is Data Analytics? - An Introduction (Full Guide)](https://www.youtube.com/watch?v=yZvFH7B6gKI) is that framing. The spine of the path is Alex The Analyst's playlist: the Intro is [FREE Data Analyst Bootcamp!!](https://www.youtube.com/watch?v=rGx1QNdYzvs\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=1), and How to plan it is [How to Become a Data Analyst in 2023 (Completely FREE!)](https://www.youtube.com/watch?v=CUBfrdDwznQ\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=2).

The first tools are spreadsheets and dashboards. Excel 21-27 in the playlist starts at [Pivot Tables in Excel | Excel Tutorials for Beginners](https://www.youtube.com/watch?v=lH7HfwUFnYA\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=22). PowerBI 33-41 starts at [How to Install Power BI | Building First Visualization | Microsoft Power BI for Beginners](https://www.youtube.com/watch?v=g0m5sEHPU-s\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=34). Tableau 23-31 comes from a Simplilearn playlist instead: [Tableau Training For Beginners | Tableau Tutorial | Tableau Tutorial For Beginners | Simplilearn](https://youtube.com/watch?v=UWT14hhjsL0\&list=PLEiEAq2VkUUKgEFXH1tBbHwq38oWYDScU\&index=24).

Dashboards sit on data that lives in a database, so SQL comes next. SQL 3-20 starts at [Installing MySQL and Creating Databases | MySQL for Beginners](https://www.youtube.com/watch?v=wgRwITQHszU\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=4).

The same notes are in [SQL](/data/engineering-intro/sql).

For SQL questions, the practice video is [Solving Easy SQL Interview Questions on Analyst Builder](https://www.youtube.com/watch?v=ZHaYOC0H5KE\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=70).

The path ends in code. Python 42-56 starts at [Installing Jupyter Notebooks/Anaconda | Python for Beginners](https://www.youtube.com/watch?v=WUeBzT43JyY\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=43), and Pandas 57-64 starts at [Reading in Files in Pandas | Python Pandas Tutorials](https://www.youtube.com/watch?v=dUpyC40cF6Q\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=58). As a single Course that covers the Python side in one sitting, freeCodeCamp.org has [Data Analysis with Python Course - Numpy, Pandas, Data Visualization](https://www.youtube.com/watch?v=GPVsHOlRBBI). Simplilearn's [Data Analytics Full Course In 5 Hours | Data Analytics For Beginners | Data Analytics | Simplilearn](https://www.youtube.com/watch?v=oQmxQh92Nh8\&list=PLEiEAq2VkUUKgEFXH1tBbHwq38oWYDScU\&index=2) is the long-form alternative, with the author's own verdict: Not sure if needed.


# Dependence and Selection

Features that move together waste capacity, and features that do not predict the target waste the model, so this page is how to measure dependence and then select. It moves from correlation versus covariance, through correlation across feature types and visualizations, into mutual information, Cramer’s coefficient, predictive power score, feature selection, and feature importance.

#### **CORRELATION VS COVARIANCE**

Measuring dependence starts with choosing a scale. Correlation is between -1 to 1, covariance is -inf to inf, units in covariance affect the scale, so correlation is preferred, it is normalized. Correlation is a measure of association. Correlation is used for bivariate analysis. It is a measure of how well the two variables are related. Covariance is also a measure of association. Covariance is a measure of the relationship between two random variables. The article these notes came from no longer opens and is kept at the end of the page.

### **CORRELATION**

With correlation chosen as the scale, the standard coefficient is Pearson. MachineLearningMastery's "How to Calculate Correlation Between Variables in Python" covers [**Pearson**](https://machinelearningmastery.com/how-to-use-correlation-to-understand-the-relationship-between-variables/) in code.

#### **CORRELATION BETWEEN FEATURE TYPES**

Pearson assumes two numeric columns, and real tables mix categorical and continuous features, so the question widens. Association vs correlation - correlation is a measure of association and a yes no question without assuming linearity. [**A great article in medium**](https://medium.com/@outside2SDs/an-overview-of-correlation-measures-between-categorical-and-continuous-variables-4c7f85610365), covering just about everything with great detail and explaining all the methods plus references, is the place to start.

For a categorical feature against the target, one quick view is heat maps for categorical vs target - groupby count per class, normalize by total count to see if you get more grouping in a certain combination of cat/target than others.

For a continuous feature against a categorical one, the tools are [**Anova**](https://www.researchgate.net/post/Which_test_do_I_use_to_estimate_the_correlation_between_an_independent_categorical_variable_and_a_dependent_continuous_variable) / [**log regression**](https://www.statalist.org/forums/forum/general-stata-discussion/general/1470627-correlation-between-continous-and-categorical-variable), with more in [**2\*,**](https://dzone.com/articles/correlation-between-categorical-and-continuous-var-1) and the [**git**](https://github.com/ShitalKat/Correlation/blob/master/Correlation%20between%20categorical%20and%20continuous%20variables.ipynb) notebook, a python code and analysis on correlation measure between categorical and continuous variable. Source 3 no longer opens and is kept at the end of the page. The same question, for numeric/[**cont vs categorical**](https://www.quora.com/How-can-I-measure-the-correlation-between-continuous-and-categorical-variables), has one rule of thumb: a high F score from anova hints about association between a feature and a target, i.e., the importance of the feature to separating the target.

The same notes are in [Distribution Transformation](/data/processing-intro/distribution-transformation).

Since that rule leans on the F score, two videos explain ANOVA itself. J David Eisenberg's Analysis of Variance (ANOVA) is Anova youtube [**1**](https://www.youtube.com/watch?v=ITf4vHhyGpc), and statisticsfun's How To Calculate and Understand Analysis of Variance (ANOVA) F Test is [**2**](https://www.youtube.com/watch?v=-yQb_ZJnFXw). The figure below goes with the continuous-versus-categorical question.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-b5a0a75f853aa1042740e88f187ad4f82fc8e69c%2Fgimg-89a80ede9194.png?alt=media" alt=""><figcaption><p>Features.</p></figcaption></figure>

The image is by multiple possible sources: [**rayhanul islam**](https://www.quora.com/How-can-I-measure-the-correlation-between-continuous-and-categorical-variables) on the same Quora question, and [**statistics for fun**](https://www.facebook.com/statneil/photos/a.787373884990868/839856346409288/?type=3).

The last pairing is categorical against categorical. Cat vs cat has many metrics - on medium; that article no longer opens and is kept at the end of the page.

#### **CORRELATION VISUALIZATION**

A table of coefficients is hard to read once there are many features, so the next step is to look at them. Feature space is that view, shown in the two figures below, both credited to Matt Britton; the original post's old address is kept at the end of the page.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-35261cf73b08efc4674a910ac98059f3802aa11c%2Fgimg-82c4a3eb690d.png?alt=media" alt=""><figcaption><p>Feature space.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-ae786a6e5b2602ccfaef26598b3fd325cb703e7b%2Fgimg-4b55c10fa38a.png?alt=media" alt=""><figcaption><p>Feature space.</p></figcaption></figure>

### **MUTUAL INFORMATION COEFFICIENT**

Correlation and its pictures still look for a particular kind of relationship; mutual information asks whether two variables share any information at all.

The same notes are in [Information Theory](/data/analytics-intro/information-theory).

The starting point is the MIC [**Paper**](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3325791/) **- we present a measure of dependence for two-variable relationships: the maximal information coefficient (MIC). MIC captures a wide range of associations both functional and not, and for functional relationships provides a score that roughly equals the coefficient of determination (R2) of the data relative to the regression function.**\\

**Computing MIC** is a grid search over the scatter plot, shown in the figure and explained under it.\\

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-011b9ba4a68d0d997e8e04202f0ddcead7625e4e%2Fgimg-66d171a4b6e1.png?alt=media" alt=""><figcaption><p>Features.</p></figcaption></figure>

**(A) For each pair (x,y), the MIC algorithm finds the x-by-y grid with the highest induced mutual information. (B) The algorithm normalizes the mutual information scores and compiles a matrix that stores, for each resolution, the best grid at that resolution and its normalized score. (C) The normalized scores form the characteristic matrix, which can be visualized as a surface; MIC corresponds to the highest point on this surface.** \\

**In this example, there are many grids that achieve the highest score. The star in (B) marks a sample grid achieving this score, and the star in (C) marks that grid's corresponding location on the surface.**\\

MIC is a paper-level measure; for day-to-day feature scoring, scikit-learn has the estimators. The [**Mutual information classifier**](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html) **- Estimate mutual information for a discrete target variable.**

**Mutual information (MI)** [**\[1\]**](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html#r50b872b699c4-1) **between two random variables is a non-negative value, which measures the dependency between the variables. It is equal to zero if and only if two random variables are independent, and higher values mean higher dependency.**

**The function relies on nonparametric methods based on entropy estimation from k-nearest neighbors distances as described in** [**\[2\]**](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html#r50b872b699c4-2) **and** [**\[3\]**](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html#r50b872b699c4-3)**. Both methods are based on the idea originally proposed in** [**\[4\]**](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html#r50b872b699c4-4)**.**\\

The same quantity also compares two labelings rather than a feature and a target. The [**MI score**](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mutual_info_score.html) **- Mutual Information between two clusterings.**

**The Mutual Information is a measure of the similarity between two labels of the same data.** \\

Raw MI rewards clusterings with more clusters, which is why two corrected versions follow. The [**Adjusted MI score**](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_mutual_info_score.html#sklearn.metrics.adjusted_mutual_info_score) **- Adjusted Mutual Information between two clusterings.**

**Adjusted Mutual Information (AMI) is an adjustment of the Mutual Information (MI) score to account for chance. It accounts for the fact that the MI is generally higher for two clusterings with a larger number of clusters, regardless of whether there is actually more information shared.**

**This metric is furthermore symmetric: switching label\_true with label\_pred will return the same score value. This can be useful to measure the agreement of two independent label assignments strategies on the same dataset when the real ground truth is not known**\\

The other correction rescales instead of adjusting for chance. The [**Normalized MI score**](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.normalized_mutual_info_score.html#sklearn.metrics.normalized_mutual_info_score) **- Normalized Mutual Information (NMI) is a normalization of the Mutual Information (MI) score to scale the results between 0 (no mutual information) and 1 (perfect correlation). In this function, mutual information is normalized by some generalized mean of H(labels\_true) and H(labels\_pred)), defined by the average\_method.**\\

### **CRAMER’S COEFFICIENT**

Mutual information handles any pair of variables; for two categorical variables there is also a coefficient built for the job, the "cat vs cat" case from earlier. [**Calculating** ](https://stackoverflow.com/questions/20892799/using-pandas-calculate-cram%C3%A9rs-coefficient-matrix)it in pandas is the Stack Overflow question on a Cramér's coefficient matrix, asked about how closely the nation and language of Wikipedia articles correlate.

##

The empty heading above is a marker that separated sections in the original notes.

### **PREDICTIVE POWER SCORE (PPS)**

Every measure so far is symmetric or type-specific, and many real relationships have a correlation of 0. Florian Wetschoreck's "RIP correlation. Introducing the Predictive Power Score" proposes a score that [**Is an asymmetric, data-type-agnostic score for predictive relationships between two columns that ranges from 0 to 1.**](https://medium.com/data-science/rip-correlation-introducing-the-predictive-power-score-3d90808b9598) The implementation is on [**github**](https://github.com/8080labs/ppscore) as Predictive Power Score (PPS) in Python.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-ed2d80cc5c0f1eddc4c0d7178eda066a7bba2818%2Fgimg-29cbbc46ce92.png?alt=media" alt=""><figcaption><p>Predictive power score examples.</p></figcaption></figure>

**Too many scenarios where the correlation is 0. This makes me wonder if I missed something… (Excerpt from the** [**image by Denis Boigelot**](https://en.wikipedia.org/wiki/Correlation_and_dependence)**)**\\

For both task types the score compares a model against a naive baseline and normalizes the result so it is never smaller than 0; only the metric changes.

**Regression**

**In case of an regression, the ppscore uses the mean absolute error (MAE) as the underlying evaluation metric (MAE\_model). The best possible score of the MAE is 0 and higher is worse. As a baseline score, we calculate the MAE of a naive model (MAE\_naive) that always predicts the median of the target column. The PPS is the result of the following normalization (and never smaller than 0):**\\

$$\text{PPS} = 1 - (\text{MAE}*{model} / \text{MAE}*{naive})$$\\

**Classification**

**If the task is a classification, we compute the weighted F1 score (wF1) as the underlying evaluation metric (F1\_model). The F1 score can be interpreted as a weighted average of the precision and recall, where an F1 score reaches its best value at 1 and worst score at 0. The relative contribution of precision and recall to the F1 score are equal. The weighted F1 takes into account the precision and recall of all classes weighted by their support as described** [**here**](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html)**. As a baseline score (F1\_naive), we calculate the weighted F1 score for a model that always predicts the most common class of the target column (F1\_most\_common) and a model that predicts random values (F1\_random). F1\_naive is set to the maximum of F1\_most\_common and F1\_random. The PPS is the result of the following normalization (and never smaller than 0):**\\

$$\text{PPS} = (F1\_{model} - F1\_{naive}) / (1 - F1\_{naive})$$\\

### **FEATURE SELECTION**

Once dependence can be scored, the practical question is which features to keep.

The same notes are in [Interpretable & Explainable AI (XAI)](/responsible-ai/interpretable-and-explainable-ai-xai) and [L1 and L2](/predictive-ml/regularization#l1-and-l2).

What follows is a series of good articles that explain about several techniques for feature selection, from a first overview through classic methods to the Relief family.

The overview comes first. A great notebook about feature correlation and manytypes of visualization, what to drop what to keep, using many feature reduction and selection methods (quite a lot actually). Its a really good intro: <https://www.kaggle.com/kanncaa1/feature-selection-and-data-visualization>. When selection is slow, [**How to parallelize feature selection on several CPUs,**](https://stackoverflow.com/questions/37037450/multi-label-feature-selection-using-sklearn) do it per label on each cpu and average the results.

Text is a common first use. [**Multi class classification, feature selection, model selection, co-feature analysis**](https://medium.com/data-science/multi-class-text-classification-with-scikit-learn-12f1e60e0a9f) is a scikit-learn text classification walkthrough for cases like tagging news by topic or products by category. [**Text analysis for sentiment, doing feature selection**](https://streamhacker.com/tag/chi-square/) is a tutorial with chi2(IG?), and StreamHacker's Text Classification for Sentiment Analysis – Stopwords and Collocations is the [**part 2 with bi-gram collocation in ntlk**](https://streamhacker.com/2010/05/24/text-classification-sentiment-analysis-stopwords-collocations/). What is collocation? - “the habitual juxtaposition of a particular word with another word or words with a frequency greater than chance.”

For the classic methods, Data Talks' Feature Selection for Scikit Learn covers [**Sklearn feature selection methods (4) - youtube**](https://www.youtube.com/watch?v=wjKvyk8xStg). The Diving into data series walks them in order: [**Univariate**](http://blog.datadive.net/selecting-good-features-part-i-univariate-selection/) selection in part I, [**Linear models and regularization,**](http://blog.datadive.net/selecting-good-features-part-ii-linear-models-and-regularization/) in part II, [**Random forests and feature ranking**](http://blog.datadive.net/selecting-good-features-part-iii-random-forests/) in part III, and [**Stability selection and recursive feature elimination (RFE).**](http://blog.datadive.net/selecting-good-features-part-iv-stability-selection-rfe-and-everything-side-by-side/) side by side in part IV. The random forest part pairs with a tuning note, [**Random Search for focus and only then grid search for Random Forest**](https://medium.com/data-science/hyperparameter-tuning-the-random-forest-in-python-using-scikit-learn-28d2aa77dd74), with the matching Improving Random Forest Part 2 notebook from WillKoehrsen/Machine-Learning-Projects as [**code**](https://github.com/WillKoehrsen/Machine-Learning-Projects/blob/master/random_forest_explained/Improving%20Random%20Forest%20Part%202.ipynb). These selectors are wrapper methods in sklearn for the purpose of feature selection; see [**RFE in sklearn**](http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html). A kernel-based alternative is C.K. Wolfe's [**Kernel feature selection via conditional covariance minimization**](http://bair.berkeley.edu/blog/2018/01/23/kernels/).

Several sources package the checks into one place. The [**Github class that does the following**](https://medium.com/data-science/a-feature-selection-tool-for-machine-learning-in-python-b64dd23710f0) finds:

* **Features with a high percentage of missing values**
* **Collinear (highly correlated) features**
* **Features with zero importance in a tree-based model**
* **Features with low importance**
* **Features with a single unique value**

MachineLearningMastery's Feature Selection For Machine Learning in Python, the [**Machinelearning mastery on FS**](https://machinelearningmastery.com/feature-selection-machine-learning-python/), covers four methods:

* **Univariate Selection.**
* **Recursive Feature Elimination.**
* **Principle Component Analysis.**
* **Feature Importance.**

The classes in the sklearn.feature\_selection module can be used for feature selection or dimensionality reduction on sample sets, either to improve estimators’ accuracy scores or to boost their performance. The [**Sklearn tutorial on FS:**](http://scikit-learn.org/stable/modules/feature_selection.html) lists:

* **Low variance**
* **Univariate kbest**
* **RFE**
* **selectFromModel using \_coef \_important\_features**
* **Linear models with L1 (svm recommended L2)**
* **Tree based importance**

[**A complete overview of many methods**](https://www.analyticsvidhya.com/blog/2016/12/introduction-to-feature-selection-methods-with-an-example-or-how-to-select-the-right-variables/) is an introduction to feature selection methods with an example, and it separates reduction, filter selection, and wrapper methods:

* **(reduction) LDA: Linear discriminant analysis is used to find a linear combination of features that characterizes or separates two or more classes (or levels) of a categorical variable.**
* **(selection) ANOVA: ANOVA stands for Analysis of variance. It is similar to LDA except for the fact that it is operated using one or more categorical independent features and one continuous dependent feature. It provides a statistical test of whether the means of several groups are equal or not.**

The same notes are in [Distribution Transformation](/data/processing-intro/distribution-transformation).

* **(Selection) Chi-Square: It is a is a statistical test applied to the groups of categorical features to evaluate the likelihood of correlation or association between them using their frequency distribution.**
* **Wrapper methods:**
* **Forward Selection: Forward selection is an iterative method in which we start with having no feature in the model. In each iteration, we keep adding the feature which best improves our model till an addition of a new variable does not improve the performance of the model.**
* **Backward Elimination: In backward elimination, we start with all the features and removes the least significant feature at each iteration which improves the performance of the model. We repeat this until no improvement is observed on removal of features.**
* **Recursive Feature elimination: It is a greedy optimization algorithm which aims to find the best performing feature subset. It repeatedly creates models and keeps aside the best or the worst performing feature at each iteration. It constructs the next model with the left features until all the features are exhausted. It then ranks the features based on the order of their elimination.**

Wrapper methods retrain the model many times; Relief scores features from distances instead. Yash Dagli's [**Relief**](https://medium.com/@yashdagli98/feature-selection-using-relief-algorithms-with-python-example-3c2006e18f83) article, with implementations in [**GIT**](https://github.com/GrantRVD/ReliefF) and [**git2**](https://pypi.org/project/ReliefF/#description), describes a new family of feature selection trying to optimize the distance of two samples from the selected one, one which should be closer the other farther. The weight update is quoted from that article:

**“The weight updation of attributes works on a simple idea (line 6). That if instance Rᵢ and H have different value (i.e the diff value is large), that means that attribute separates two instance with the same class which is not desirable, thus we reduce the attributes weight. On the other hand, if the instance Rᵢ and M have different value, that means the attribute separates the two instance with different class, which is desirable.”**

Relief also ships in larger libraries. [**Scikit-feature (includes relief)**](https://github.com/chappers/scikit-feature) is feature selection with scikit-learn, forked from [**this**](https://github.com/jundongl/scikit-feature/tree/master/skfeature) open-source feature selection repository in python, whose [**(docs)**](http://featureselection.asu.edu/algorithms.php) page is linked beside it. [**Scikit-rebate (based on relief)**](https://github.com/EpistasisLab/scikit-rebate) is a scikit-learn-compatible Python implementation of ReBATE, a suite of Relief-based feature selection algorithms.

Selection can also go back to the information measures from earlier on the page. [**Feature selection using entropy, information gain, mutual information and … in sklearn.**](https://gist.github.com/GaelVaroquaux/ead9898bd3c973c40429) is a gist for estimating entropy and mutual information with scikit-learn, and the theory behind it is in [**Entropy, mutual information and KL Divergence by AurelienGeron**](https://www.techleer.com/articles/496-a-short-introduction-to-entropy-cross-entropy-and-kl-divergence-aurelien-geron/).

### **FEATURE IMPORTANCE**

Selection decides which features stay; importance asks how much each kept feature actually drives the prediction.

The same notes are in [Lime](/responsible-ai/interpretable-and-explainable-ai-xai#lime) and [Shap](/responsible-ai/interpretable-and-explainable-ai-xai#shap).

Note: the point about lime below is used for explainability, please also check that topic.

The first read is Eryk Lewinson's Explaining Feature Importance by example of a Random Forest, [**Using RF and other methods, really good**](https://medium.com/data-science/explaining-feature-importance-by-example-of-a-random-forest-d9166011959e). Rankings are not the same as measured impact, which is the point of [**Non parametric feature impact and importance**](https://arxiv.org/abs/2006.04750) **- while there are nonparametric feature selection algorithms, they typically provide feature rankings, rather than measures of impact or importance.In this paper, we give mathematical definitions of feature impact and importance, derived from partial dependence curves, that operate directly on the data.**

LIME goes one step further and explains single predictions. The "Why Should I Trust You?" [**Paper**](https://arxiv.org/abs/1602.04938) **(**[**pdf**](https://arxiv.org/pdf/1602.04938.pdf)**,** [**blog post**](https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime)**): (**[**GITHUB**](https://github.com/marcotcr/lime/blob/master/README.md)**) how to "explain the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction."**\
\
**they want to understand the reasons behind the predictions, it’s a new field that says that many 'feature importance' measures shouldn’t be used. i.e., in a linear regression model, a feature can have an importance rank of 50 (for example), in a comparative model where you duplicate that feature 50 times, each one will have 1/50 importance and won’t be selected for the top K, but it will still be one of the most important features. so new methods needs to be developed to understand feature importance. this one has git code as well.**

To try it, there are **Several github notebook examples:** [**binary case**](https://marcotcr.github.io/lime/tutorials/Lime%20-%20basic%20usage%2C%20two%20class%20case.html)**,** [**multi class**](https://marcotcr.github.io/lime/tutorials/Lime%20-%20multiclass.html)**,** [**cont and cat features**](https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20continuous%20and%20categorical%20features.html)**, there are many more for images in the github link.**\\

The authors' own description of the idea, and the figure it refers to, close the page:

**“Intuitively, an explanation is a local linear approximation of the model's behaviour. While the model may be very complex globally, it is easier to approximate it around the vicinity of a particular instance. While treating the model as a black box, we perturb the instance we want to explain and learn a sparse linear model around it, as an explanation. The figure below illustrates the intuition for this procedure. The model's decision function is represented by the blue/pink background, and is clearly nonlinear. The bright red cross is the instance being explained (let's call it X). We sample instances around X, and weight them according to their proximity to X (weight here is indicated by size). We then learn a linear model (dashed line) that approximates the model well in the vicinity of X, but not necessarily globally. For more information, read our paper, or take a look at this blog post.”**\
\\

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-d05a22ee684635203e1afcc2d00a2193a5d19221%2Fgimg-42bef46c1afd.png?alt=media" alt=""><figcaption><p>Features.</p></figcaption></figure>

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

\- Correlation is between -1 to 1, covariance is -inf to inf, units in covariance affect the scale, so correlation is preferred, it is normalized.. This address no longer opens: <https://towardsdatascience.com/correlation-coefficient-clearly-explained-f034d00b66ac> - Feature space. This address no longer opens: <https://towardsdatascience.com/escape-the-correlation-matrix-into-feature-space-4d71c51f25e5> - Cat vs cat\*\*, many metrics - on medium\*\*. This address no longer opens: <https://towardsdatascience.com/the-search-for-categorical-correlation-a1cf7f1888c9> - 3. This address no longer opens: <https://www.edvancer.in/DESCRIPTIVE+STATISTICS+FOR+DATA+SCIENCE-2> - Github class that does the following. This address no longer opens: <https://towardsdatascience.com/a-feature-selection-tool-for-machine-learning-in-python-b64dd23710f0>


# Next

Next is Data Engineering. Analysis that stays on a laptop does not serve the product. Reading a distribution, a dependence, and an information measure only pays off once the dataset lives in a system other people can run.


# Data Engineering

The dataset has to live in a system other people can run. After this chapter the reader can place SQL, storage, quality, lineage, and ownership on one path from source to table.

The path starts with asking and storing. A warehouse only answers if you can ask it in SQL, so [SQL](/data/engineering-intro/sql) comes first: command groups, keys, indexes, and how sharding differs from partitioning. [Tools](/data/engineering-intro/tools) is the shelf for change capture, lakes, pipelines, BI, integration, and quality checks, opened by the job each group is for. Data then has to live somewhere queryable, and [Databases](/data/engineering-intro/lakes-and-warehouses) is the choice between lake, warehouse, table format, or a specialized engine. Because storage choices and models decide how a warehouse can be queried later, [Database Modeling](/data/engineering-intro/database-architecture-and-modeling) follows with warehouse modeling, Data Vault, data fabric, and virtualization.

With storage in place, the chapter turns to the job and the system around it. [Data Engineering](/data/engineering-intro/data-engineering) is the job that makes data usable, which is not the same job as writing product software. A platform needs an architecture before the warehouses and pipelines stack up, and [Data Architecture](/data/engineering-intro/data-architecture) is the short list of data-platform architectures and domain-driven design for data. [Data Platforms](/data/engineering-intro/data-platforms) is where compute and storage meet so teams can run jobs without reinventing the lake each time. [Data Patterns](/data/engineering-intro/patterns) gives the rule for what history looks like when a row changes over time.

Once tables flow, they have to be trusted. Bad rows break models and dashboards, so [Data Quality](/data/engineering-intro/data-quality) defines and tests quality instead of assuming it. Pipelines fail quietly unless someone watches freshness, volume, and schema drift, which is [Data Observability](/data/engineering-intro/data-observability). When a number is wrong, someone has to walk the path the data took, and that is [Data Lineage](/data/engineering-intro/data-lineage). Producers and consumers need a promise about the shape and meaning of a table, the [Data Contract](/data/engineering-intro/data-contract).

Trust then becomes ownership. [Data Governance](/data/engineering-intro/data-governance) is about who may change the data and what "good" means across teams. A central data team can become the bottleneck, so [Data Mesh](/data/engineering-intro/data-mesh) treats domains as owners of data products, and [Data Product](/data/engineering-intro/data-product) is the work of treating data as something with users and owners, since a platform nobody adopts is not a product.

The chapter ends with the organization that pays for all of it. [Data Strategy](/data/engineering-intro/data-strategy) says what data work is for before the company buys another tool. [Data Vision](/data/engineering-intro/data-vision) says what success looks like before the org chart is drawn. [Data Teams](/data/engineering-intro/data-teams) is the team shape that can ship analytics. [Data KPIs](/data/engineering-intro/data-kpis) measures the return on data spend, not only pipeline uptime. [Data Program Management](/data/engineering-intro/data-program-management) is the roles and definitions that keep projects from drifting.


# Data Science Tools

The workbench for data science is Python and the libraries around it, plus a notebook and a way to keep environments from colliding. The page starts with the language itself, its async, clean-code, and virtual-environment habits, then moves to Jupyter, SciPy, NumPy, Pandas and EDA, and closes with Git.

## Python

Everything else on the bench is written in Python, so the first habit is writing the language well. [How to use better OOP in python.](https://hackernoon.com/improve-your-python-python-classes-and-object-oriented-programming-d09ff461168d) is "Improve Your Python: Python Classes and Object Oriented Programming" on HackerNoon, the author's most popular post and an accessible introduction to classes and OOP. [Best practices programming python classes - a great lecture.](https://www.youtube.com/watch?v=HTLu2DFOdTg) is "Python's Class Development Toolkit", published by Next Day Video.

Once the code is organized, the next questions are what it costs and what it promises. [How to know pip packages size](https://stackoverflow.com/questions/34266159/how-to-see-pip-package-sizes-installed) is the question of how to list every installed pip package with the disk space each one takes up. [Python type checking tutorial](https://medium.com/@ageitgey/learn-how-to-use-static-type-checking-in-python-3-6-in-10-minutes-12c86d72677b) is Adam Geitgey on static type checking in Python 3.6, the answer to the complaint that Python variables are dynamically typed. For scripts other people run, [Import click - command line interface](https://zetcode.com/python/click/) is Jan Bodnar's tutorial on building command line interfaces with the click module, an alternative to optparse and argparse.

Speed is the last Python question, and it starts with a distinction. [Concurrency vs Parallelism (great)](https://stackoverflow.com/questions/1050222/what-is-the-difference-between-concurrency-and-parallelism) is the question that separates the two. [Async in python](https://medium.com/velotio-perspectives/an-introduction-to-asynchronous-programming-in-python-af0189a88bbb) introduces asynchronous programming as a unit of work that runs separately from the main thread and reports back when it completes or fails. [Coroutines vs futures](https://stackoverflow.com/questions/34753401/difference-between-coroutine-and-future-task-in-python-3-5) is the difference between a coroutine and a future or task in Python 3.5. Three notes that went with it, coroutines generators async wait, an intro to concurrent,futures, and future task event loop, used to link here; they are kept at the end of the page.

### Async io

The async note above needs one library to make it concrete, and that is asyncio. The [Intro](https://realpython.com/lessons/what-asyncio/) is Real Python's lesson on what Async IO is, how it differs from multiprocessing, and how requests are dispatched and responses handled through the event loop. The [complete](https://realpython.com/async-io-python/) version is the full Real Python walk-through that follows the lesson.

### Clean code

Fast code still has to be readable by the next person. [Clean code in python git](https://github.com/zedr/clean-code-python) is the repository of Clean Code concepts adapted for Python, :bathtub: included. [About the book](https://medium.com/@m_mcclarty/tech-book-talk-clean-code-in-python-aa2c92c6564f) is the book talk that goes with clean code in Python.

### Virtual Environments

Clean code breaks anyway when two projects need different versions of the same package, which is why the environment has to be isolated. The number of tools is the confusing part, so the first stop is [stack overflow on pyenv / venv / etc](https://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe), the question of how Python 3.3's standard-library venv differs from every other package matching the (py)?(v|virtual|pip)?env pattern. [Guide to pyenv & pyenv virtualenv](https://medium.com/swlh/a-guide-to-python-virtual-environments-8af34aa106ac) is Ray Johns on nearly-painless virtual Python installations with pyenv, pyenv-virtualenv, and Homebrew. [Managing virtual env with pyenv](https://medium.com/data-science/managing-virtual-environment-with-pyenv-ae6f3fb835f8) is for the point where tens of environments for different projects become daunting, and pyenv streamlines creating, managing, and activating them.

The shortest advice is: Just use venv. The article behind that advice no longer opens and is kept at the end of the page. The same stack overflow question is also the [Summary on all the \*envs](https://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe), and [A really good primer on virtual environments](https://realpython.com/python-virtual-environments-a-primer/) is Real Python's primer. [Introduction to venv](http://cewing.github.io/training.python_web/html/presentations/venv_intro.html) is the venv presentation from Internet Programming with Python.

Pipenv is the tool that folds the environment and the dependencies into one workflow. [Pipenv](https://pipenv.readthedocs.io/en/latest/) is its documentation, "Python Development Workflow for Humans". [A great intro to pipenv](https://realpython.com/pipenv-guide/) is Real Python's guide to the packaging tool, and [A complementary to pipenv above](https://robots.thoughtbot.com/how-to-manage-your-python-projects-with-pipenv) is Murtaza Gulamali on using Pipenv to simplify dependency management in Python projects. For the side-by-side, the [Comparison between all \*env](https://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe) is that same stack overflow thread again.

The environment then has to reach the notebook. Two sources did that work: pyenv, virtualenv and using them with Jupyter, a make sense tutorial and instructions on how to use all, and Create isolated Jupyter ipython kernels with pyenv and virtualenv by alfredo motta. Both addresses are kept at the end of the page. What still opens is [Jupyter Notebook in a virtual env](https://medium.com/data-science/jupyter-notebooks-i-getting-started-with-jupyter-notebooks-f529449797d2), which shows how to install a kernelspec so Jupyter Notebook can reach your Python data science virtual environments.

#### PYENV

Of all the tools above, pyenv is the one that also switches the Python version itself. [Installing pyenv](https://bgasparotto.com/install-pyenv-ubuntu-debian) is the install on Ubuntu and other Debian distributions, so you can quickly switch between Python versions. [Intro to pyenv](https://realpython.com/intro-to-pyenv/) is Real Python on managing multiple Python versions with it. [Pyenv tutorial and finding where it is](https://anil.io/blog/python/pyenv/using-pyenv-to-install-multiple-python-versions-tox/) covers installing, managing, and switching between versions, most commonly to test code across multiple Python environments. On a Mac the system Python gets in the way: [Pyenv override system python on mac](https://github.com/pyenv/pyenv/issues/660) is the pyenv issue about being unable to override the Python 2.7.10 that came with OSX 10.11.5 after a Homebrew install. The last step is pyenv virtualenv, which joins pyenv to the environment notes above.

## Jupyter

With the environment in place, the notebook is where the work actually happens. The same notes are in [Docker](/ai-engineering/full-stack-and-ops-1/docker) and [Docker for data science](/ai-engineering/full-stack-and-ops#docker-for-data-science).

A notebook needs hardware first. [Cloud GPUS cheap](https://www.paperspace.com/gradient) is the MLops platform page, now on DigitalOcean, for building and scaling an AI project on GPUs. Inside the notebook the daily habits come next. [Importing a notebook as a module](http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Importing%20Notebooks.html) is the Jupyter documentation for importing notebooks as modules. The important [colaboratory commands for jupytr](https://medium.com/deep-learning-turkey/google-colab-free-gpu-tutorial-e113627b9f5d) are in the Google Colab free GPU tutorial. Timing and profiling in Jupyter used to have its own page, kept at the end. ([Debugging in Jupyter, how?)](https://kawahara.ca/how-to-debug-a-jupyter-ipython-notebook/) - put a one liner before the code and query the variables inside a function. [28 tips n tricks for jupyter](https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/) is Celeste Grupman's list of tips, tricks, and shortcuts for turning into a Jupyter notebooks power user.

The notebook can also become the package, which is the idea of Jupyter notebooks as a module. [Nbdev](https://github.com/fastai/nbdev) is the repository for creating delightful software with Jupyter Notebooks, and its site on [fast.ai](https://nbdev.fast.ai/) puts writing, testing, documenting, and distributing packages and technical articles all in one place, your notebook. The older Nbdev announcement is kept at the end of the page. [jupytext](https://github.com/mwouts/jupytext) goes the other way and stores Jupyter Notebooks as Markdown Documents, Julia, Python or R scripts.

The environment from the Python section has to be wired into Jupyter by hand. These are the steps for virtual environments in jupyter (their original page is kept at the end):

1. Enter your project directory
2. $ python -m venv projectname
3. $ source projectname/bin/activate
4. (venv) $ pip install ipykernel
5. (venv) $ ipython kernel install --user --name=projectname
6. Run jupyter notebook \* (not entirely sure how this works out when you have multiple notebook processes, can we just reuse the same server?)
7. Connect to the new server at port 8889

[Virtual env with jupyter](https://janakiev.com/til/jupyter-virtual-envs/) is the page that walks the same setup.

Inside a notebook, the first array operation that confuses people is reshape. ([how does reshape work?)](http://anie.me/numpy-reshape-transpose-theano-dimshuffle/) - a shape of (2,4,6) is like a tree of 2->4 and each one has more leaves 4->6.

As far as i can tell, reshape effectively flattens the tree and divide it again to a new tree, but the total amount of inputs needs to stay the same. 2*4*6 = 4*2*3\*2 for example

The code below shows it on a random array:

```python
import numpy
rng = numpy.random.RandomState(234)
a = rng.randn(2,3,10)
print(a.shape)
print(a)
b = numpy.reshape(a, (3,5,-1))
print(b.shape)
print(b)
```

When the laptop is not enough, the notebook moves to someone else's GPU. \*\*\* A tutorial for [Google Colaboratory - free Tesla K80 with Jup-notebook](https://www.kdnuggets.com/2018/02/google-colab-free-gpu-tutorial-tensorflow-keras-pytorch.html/2) shows deep learning development in Colab on the free Tesla K80 GPU with Keras, TensorFlow and PyTorch. [Jupyter on Amazon AWS](https://blog.keras.io/running-jupyter-notebooks-on-gpu-on-aws-a-starter-guide.html) is the Keras blog starter guide to running notebooks on GPU on AWS.

How to add extensions to jupyter: [extensions](https://codeburst.io/jupyter-notebook-tricks-for-data-science-that-enhance-your-efficiency-95f98d3adee4) is Nok Chan's Jupyter tricks for data science, updated with a table of contents, the %debug magic, and nbdime for notebook diffing. [Connecting from COLAB to MS AZURE](https://medium.com/@d.sakryukin/simple-cryptocurrency-trading-data-preparation-in-15-minutes-using-ms-azure-and-google-colab-44872b023d11) is Dmitriy Sakryukin's cryptocurrency trading data preparation in 15 minutes using MS Azure and Google Colab.

A notebook is also where a result first gets shown to someone, and that is where the dashboard tools compete. [Streamlit vs. Dash vs. Shiny vs. Voila vs. Flask vs. Jupyter](https://medium.com/data-science/streamlit-vs-dash-vs-shiny-vs-voila-vs-flask-vs-jupyter-24739ab5d569) is Markus Schmitt's comparison, written as Dash and Streamlit surged in popularity as all-in-one dashboarding solutions. Its older address is kept at the end of the page.

## SciPy

Beside the notebook, SciPy is the library you reach for when the question is a minimum or a maximum. [Optimization problems, a nice tutorial](http://scipy-lectures.org/advanced/mathematical_optimization/) is the Scipy lecture notes chapter on mathematical optimization, finding minima of functions. For the simpler case, [Minima / maxima](https://stackoverflow.com/questions/4624970/finding-local-maxima-minima-with-numpy-in-a-1d-numpy-array) asks for a numpy or scipy function that finds local maxima and minima in a 1D array instead of checking nearest neighbours by hand.

## NumPy

SciPy runs on NumPy arrays, and the speed of both comes from not writing loops. [Using numpy efficiently](https://speakerdeck.com/cournape/using-numpy-efficiently) - explaining why vectors work faster, through broadcasting, indexing, and basic internals. [Fast vector calculation, a benchmark](https://medium.com/data-science/data-science-with-python-turn-your-conditional-loops-to-numpy-vectors-9484ff9c622e) between list, map, vectorize. Vectorize wins. The idea is to use vectorize and a function that does something that may involve if conditions on a vector, and do it as fast as possible.

## Pandas

NumPy gives fast arrays, and Pandas puts names, rows, and columns on top of them, which is where most exploratory work lives. The same notes are in [Multi CPU Processing](/data/engineering-intro/multi-cpu-processing).

The starting point is a [Great introductory tutorial](http://nikgrozev.com/2015/12/27/pandas-in-jupyter-quickstart-and-useful-snippets/#loading_csv_files) about using pandas, loading, loading from zip, seeing the table’s features, accessing rows & columns, boolean operations, calculating on a whole row/column with a simple function and on two columns even, dealing with time/date parsing. Once the basics are comfortable, reshaping is next: [Visualizing pandas pivoting and reshaping functions by Jay Alammar](http://jalammar.github.io/visualizing-pandas-pivoting-and-reshaping/) picks up after 10 Minutes to pandas and shows the Reshaping and Pivot Tables functions visually. [How to beautify pandas dataframe using html display](https://stackoverflow.com/questions/26873127/show-dataframe-as-table-in-ipython-notebook) is the question of why only the last DataFrame prints as a table in an iPython notebook and how to force both.

Speed is the next problem. [Speeding up pandas](https://realpython.com/fast-flexible-pandas/) is Real Python on making pandas projects fast, flexible, easy and intuitive. [The fastest way to select rows by columns, by using masked values](https://stackoverflow.com/questions/17071871/select-rows-from-a-dataframe-based-on-values-in-a-column-in-pandas) (benchmarked): is the pandas version of SQL's `SELECT * FROM table WHERE column_name = some_value`, and the winner is:

```
def mask_with_values(df): mask = df['A'].values == 'foo' return df[mask]
```

Past that, the answer is parallelism, pools, threads, dask; that source is kept at the end of the page. [Accessing dataframe rows, columns and cells](http://pythonhow.com/accessing-dataframe-columns-rows-and-cells/)- by name, by index, by python methods. When you do have to go row by row, [Looping through pandas](https://medium.com/swlh/how-to-efficiently-loop-through-pandas-dataframe-660e4660125d) is for looping through each row of a large DataFrame with complex computation. How to inject headers into a headless CSV file used to be linked here and is kept at the end.

Time is the column that needs the most care. [Dealing with time series](http://pandas.pydata.org/pandas-docs/stable/timeseries.html) in pandas, starts from the pandas date functionality documentation. To [Create a new column](https://stackoverflow.com/questions/25570147/add-new-column-based-on-boolean-values-in-a-different-column) based on a (boolean or not) column and calculation: there are three ways, using python (map), using numpy, or using a function (not as pretty). Given a DataFrame, the [shift](http://machinelearningmastery.com/convert-time-series-supervised-learning-problem-python/)() function can be used to create copies of columns that are pushed forward (rows of NaN values added to the front) or pulled back (rows of NaN values added to the end):

1. df\['t'] = \[x for x in range(10)]
2. df\['t-1'] = df\['t'].shift(1)
3. df\['t-1'] = df\['t'].shift(-1)

[Row and column sum in pandas and numpy](http://blog.mathandpencil.com/column-and-row-sums) is the Math and Pencil note on column and row sums in Pandas and Numpy.

A dataframe that loads is not yet a dataframe you can trust, so validation comes next. [Dataframe Validation In Python](https://www.youtube.com/watch?time_continue=905\&v=1fHGXOfiDO0) - A Practical Introduction - Yotam Perkal - PyCon Israel 2018. In this talk, I will present the problem and give a practical overview (accompanied by Jupyter Notebook code examples) of three libraries that aim to address it: Voluptuous - Which uses Schema definitions in order to validate data \[<https://github.com/alecthomas/voluptuous>] Engarde - A lightweight way to explicitly state your assumptions about the data and check that they're actually true \[<https://github.com/TomAugspurger/engarde>] \* TDDA - Test Driven Data Analysis \[ <https://github.com/tdda/tdda>]. By the end of this talk, you will understand the Importance of data validation and get a sense of how to integrate data validation principles as part of the ML pipeline. Those three repositories are Voluptuous, a Python data validation library despite the name; engarde, a library for defensive data analysis; and tdda, test-driven data analysis functions.

The same notes are in [Data & Model Tests](/evals/data-and-model-tests).

With valid data, the everyday operations are iterating, grouping, and flattening. [Stop using itterows](https://medium.com/@rtjeannier/pandas-101-cont-9d061cb73bfc) use apply, because relying on the convenience of iterrows is a terrible habit when iterating over a DataFrame. [(great) Group and Aggregate by One or More Columns in Pandas](https://jamesrledoux.com/code/group-by-aggregate-pandas) is by a data scientist and armchair sabermetrician. [Pandas Groupby: Summarising, Aggregating, and Grouping data in Python](https://www.shanelynn.ie/summarising-aggregation-and-grouping-data-in-python-pandas/#applying-multiple-functions-to-columns-in-groups) shows grouping and aggregation with `groupby()` and `agg()`, applying max, min, count, and distinct to groups. The 25 pandas functions you didnt know about are kept at the end of the page. For nested data, [json\_normalize()](https://medium.com/data-science/all-pandas-json-normalize-you-should-know-for-flattening-json-13eae1dfb7dd) is the pandas function for flattening JSON.

### Exploratory Data Analysis (EDA)

Once the dataframe is clean, EDA is the first look at what is in it, and these tools do that look in one line. [Pandas summary](https://github.com/mouradmourafiq/pandas-summary) now points to Polyaxon's traceml, an engine for AI/ML/Data tracking, visualization, explainability, drift detection, and dashboards. [Pandas html profiling](https://github.com/pandas-profiling/pandas-profiling) is one line of code data quality profiling and exploratory data analysis for Pandas and Spark DataFrames. [Sweetviz](https://github.com/fbdesignpro/sweetviz) - "Sweetviz is an open-source Python library that generates beautiful, high-density visualizations to kickstart EDA (Exploratory Data Analysis) with just two lines of code. Output is a fully self-contained HTML application.

The system is built around quickly visualizing target values and comparing datasets. Its goal is to help quick analysis of target characteristics, training vs testing data, and other such data characterization tasks."

The figure below is Sweetviz output, credited to Sweetviz.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Mgd48oS5_duTKOVE_Et%2F-MkGjw9tWuaKDmyma1jJ%2F-MkGmutJJg5_dP48jIoO%2Fimage.png?alt=media&amp;token=3e6a91e9-9392-4174-9b70-980f499db754" alt=""><figcaption><p>Sweetviz EDA output.</p></figcaption></figure>

## GIT / Bitbucket

Closing the workbench, the notebooks, environments, and datasets above all need version control. [understanding git](https://learngitbranching.js.org/) is Peter Cottle's interactive Git visualization tool to educate and challenge. Before anything is committed, [pre-commit](https://pre-commit.com/) is the project to check. When history needs fixing, [Rewrite git history, all the commands](https://www.youtube.com/watch?v=ElRzTuYln0M) is The Modern Coder on how to amend, reword, delete, reorder, squash and split.

Datasets are too big for plain Git, which is what LFS is for. [Installing git LFS](https://askubuntu.com/questions/799341/how-to-install-git-lfs-on-ubuntu-16-04) is the question of installing git-lfs on Ubuntu when only Debian packages are offered. [Use git lfs](https://confluence.atlassian.com/bitbucket/use-git-lfs-with-bitbucket-828781636.html) is Atlassian's support page for Git LFS with Bitbucket Cloud. [Download git-lfs](https://git-lfs.github.com/) is Git Large File Storage, which replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git while the contents live on a remote server like GitHub.com or GitHub Enterprise. With many branches in flight, [Git wip](https://carolynvanslyck.com/blog/2020/12/git-wip/) creates a git wip command that lists your branches and when you last changed them, answering "what the heck was I just doing?"

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-88c647b3d83b0a1ba486d493f3ea1f7d22c7af35%2Fgimg-134013188da4.png?alt=media" alt=""><figcaption><p>Git WIP illustration.</p></figcaption></figure>

The git wip illustration is by [Carolyn Van Slyck](https://carolynvanslyck.com/)

A few notes from earlier sections also had towardsdatascience.com addresses, kept here as they were. For managing virtual env with pyenv, <https://towardsdatascience.com/managing-virtual-environment-with-pyenv-ae6f3fb835f8> now opens a page titled Convolutional Neural Networks, Explained, by Mayank Mishra, so the medium.com copy in the Virtual Environments section is the one to read. Jupyter Notebook in a virtual env by Christine Egan is at <https://towardsdatascience.com/jupyter-notebooks-i-getting-started-with-jupyter-notebooks-f529449797d2>. Fast vector calculation, a benchmark, is Tirthajyoti Sarkar's point that it pays to even vectorize conditional loops for speeding up the overall data transformation: <https://towardsdatascience.com/data-science-with-python-turn-your-conditional-loops-to-numpy-vectors-9484ff9c622e>. (good) Pandas time series manipulation covers data analysis and manipulation, plotting, resampling, and rolling: <https://towardsdatascience.com/practical-guide-for-time-series-analysis-with-pandas-196b8b46858f>

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Coroutines generators async wait. This address no longer opens: <https://masnun.com/2015/11/13/python-generators-coroutines-native-coroutines-and-async-await.html>
* Intro to concurrent,futures. This address no longer opens: <http://masnun.com/2016/03/29/python-a-quick-introduction-to-the-concurrent-futures-module.html>
* Future task event loop. This address no longer opens: <https://masnun.com/2015/11/20/python-asyncio-future-task-and-the-event-loop.html>
* Just use venv. This address no longer opens: <https://towardsdatascience.com/all-you-need-to-know-about-python-virtual-environments-9b4aae690f97>
* pyenv, virtualenv and using them with Jupyter. This address no longer opens: <https://albertauyeung.github.io/2020/08/17/pyenv-jupyter.html/>
* Create isolated Jupyter ipython kernels with pyenv and virtualenv by alfredo motta. This address no longer opens: <https://www.alfredo.motta.name/create-isolated-jupyter-ipython-kernels-with-pyenv-and-virtualenv/>
* Timing and profiling in Jupyter. This address no longer opens: <http://pynash.org/2013/03/06/timing-and-profiling/>
* Nbdev on fast.ai. This address no longer opens: <https://www.fast.ai/2019/12/02/nbdev/>
* Virtual environments in jupyter. This address no longer opens: <https://anbasile.github.io/programming/2017/06/25/jupyter-venv/>
* Streamlit vs. Dash vs. Shiny vs. Voila vs. Flask vs. Jupyter. This address no longer opens: <https://towardsdatascience.com/streamlit-vs-dash-vs-shiny-vs-voila-vs-flask-vs-jupyter-24739ab5d569>
* Parallelism, pools, threads, dask. This address no longer opens: <https://towardsdatascience.com/speed-up-your-algorithms-part-3-parallelization-4d95c0888748#7e6e>
* How to inject headers into a headless CSV file. This address no longer opens: <http://pythonforengineers.com/introduction-to-pandas/>
* pandas function you didnt know about. This address no longer opens: <https://towardsdatascience.com/25-pandas-functions-you-didnt-know-existed-p-guarantee-0-8-1a05dcaad5d0>
* Using resample. This address no longer opens: <https://towardsdatascience.com/using-the-pandas-resample-function-a231144194c4>
* Basic TS manipulation. This address no longer opens: <https://towardsdatascience.com/basic-time-series-manipulation-with-pandas-4432afee64ea>
* Functional api for sk learn. This address no longer opens: <https://scikit-lego.readthedocs.io/en/latest/preprocessing.html>


# Multi CPU Processing

One process is not enough once NumPy and Pandas jobs get large, so this page is how those same libraries use more than one CPU. It moves from NumPy and Pandas on more than one process, through Dask, then async. The same notes are in [Pandas](/data/engineering-intro/data-science-tools#pandas).

The first step is to keep the libraries you already use and give them more processes. [Numpy](https://gitlab.com/tenzing/shared-array) on multi process, and [how to use it.](https://medium.com/analytics-vidhya/multiprocessing-for-data-scientists-in-python-427b2ff93af1) are the SharedArray project on GitLab and Sebastian Theiler's multiprocessing for data scientists in Python. For dataframes, [Pandas on multi process](https://github.com/nalepae/pandarallel) is pandarallel, a simple and efficient tool to parallelize Pandas operations on all available CPUs.

When one machine's processes are still not enough, Dask is the library that schedules the work. [Dask](https://docs.dask.org/en/latest/) is the Dask documentation, and the Dask youtube channel has the [intros](https://www.youtube.com/channel/UCj9eavqmvwaCyKhIlu2GaoA). Diagnostic [dashboards](https://www.youtube.com/watch?v=N_GqzcuGLCY) is the Dask Dashboard walkthrough by Matthew Rocklin, and [Distributed sklearn](https://www.youtube.com/watch?v=5Zf6DQaf7jk) is distributed scikit learn by Tom Augspurger.

Dask is not the only way to speed up an apply, and the comparison of Dask vs swifter vs vectorize (its source is kept at the end of the page) comes down to three notes:

1. Dask is dask
2. Swifter will attempt to understand if dask or pandas apply should be used, looks like its using multi cpu so it may not be just using dask on the backend?
3. Vectorize is just another option

Outside dataframes, the same idea applies to any Python job. [Multi process cpu example](https://datascience.blog.wzb.eu/2017/06/19/speeding-up-nltk-with-parallel-processing/) is the WZB Data Science Blog on speeding up NLTK with parallel processing. [Medium on MP, using MP pool, Ray etc.](https://medium.com/distributed-computing-with-ray/how-to-scale-python-multiprocessing-to-a-cluster-with-one-line-of-code-d19f242f60ff) is Edward Oakes showing how `multiprocessing.Pool` can be scaled from a single machine to a cluster.

Processes are one answer; the last one is not to wait at all. [Async (multi process/thread/coroutines/asyncio)](https://medium.com/velotio-perspectives/an-introduction-to-asynchronous-programming-in-python-af0189a88bbb) introduces asynchronous programming, where a unit of work runs separately from the main thread and notifies it when it completes or fails.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Dask vs swifter vs vectorize. This address no longer opens: <https://gdcoder.com/speed-up-pandas-apply-function-using-dask-or-swifter-tutorial/>


# SQL

A warehouse only answers if you can ask it in SQL, so this page is the command groups, keys, indexes, and how sharding differs from partitioning. It stays one beat: asking a warehouse a question, starting with what kinds of commands exist, then the keys and indexes that make a question fast, then how a large table is split.

The same notes are in [Course](/data/analytics-intro/data-analytics#course).

Every SQL statement belongs to one of five command groups, and knowing the group tells you whether you are describing, reading, changing, permitting, or committing data. The (very good) [DDL, DQL, DML, DCL and TCL](https://www.geeksforgeeks.org/sql-ddl-dql-dml-dcl-tcl-commands/), by GeeksforGeeks, the all-in-one learning portal, lists them with examples:

* DDL, Data Definition Language: create, drop, alter, truncate.
* DQL, Data Query Language: select.
* DML, Data Manipulation Language: update, insert, delete, lock.
* DCL, Data Control Language: grant, revoke.
* TCL, Transaction Control Language: commit, rollback, savepoint, set transaction.

With the groups named, the fastest way to see them in use is a crash course. [Introduction, index, keys, joins, aliases, and so on](https://www.youtube.com/watch?v=nWeW3sCmD2k) is Traversy Media's SQL Crash Course, beginner to intermediate, and the [newer](https://www.youtube.com/watch?v=9ylj9NR0Lcg) one is their MySQL Crash Course. The [SQL cheat sheet](https://gist.github.com/bradtraversy/c831baaad44343cc945e76c2e30927b3) is the MySQL Cheat Sheet gist to keep beside it.

The course touches keys and indexes; they deserve their own look because they decide how tables connect and how fast a question comes back. [Primary key](https://www.eukhost.com/blog/webhosting/whats-the-purpose-use-primary-foreign-keys/) is the purpose of primary and foreign keys. [Foreign key, a key constraint that is included in the primary key's allowed values](https://www.1keydata.com/sql/sql-foreign-key.html) shows how foreign key constraints link tables and enforce referential integrity, with CREATE TABLE and ALTER TABLE syntax for MySQL, Oracle, and SQL Server and composite key examples. [Index, that is, a book index for fast reading](https://www.tutorialspoint.com/sql/sql-indexes.htm) describes indexes as special lookup tables holding pointers to the data, so records are located faster.

Indexes help until the table itself is too large for one place. [Sharding vs partitioning](https://planetscale.com/learn/articles/sharding-vs-partitioning-whats-the-difference) is PlanetScale on the two common ways to improve the performance, manageability, and availability of larger databases, and how they differ.


# Tools

Data work needs a shelf of tools for change capture, lakes, pipelines, BI, integration, and quality checks. This page is that tool shelf, opened by the job each group is for, in the order data moves: captured at the source, landed in a lake, transformed, shown in BI, moved between systems, and finally tested.

The same notes are in [CDC](/data/processing-intro/data-pipelines#cdc). The same notes are in [Data lake table formats](/data/engineering-intro/lakes-and-warehouses#data-lake-table-formats). The same notes are in [DBT](/data/engineering-intro/data-quality#dbt) and [Snowflake](/data/engineering-intro/lakes-and-warehouses#snowflake). The same notes are in [Data Validation](/data/engineering-intro/data-quality#data-validation).

Data starts as changes in someone else's database, so the first tool captures those changes. [Debezium](https://debezium.io/) — an open source distributed platform for change data capture.

The captured changes have to land somewhere that can take a stream and still be queried in batch. [Hudi](https://hudi.apache.org/) is Apache Hudi, an open source data lake platform, described in its own words:

> Hudi is a rich platform to build streaming data lakes with incremental data pipelines on a self-managing database layer, while being optimized for lake engines and regular batch processing.

[Upsolver](https://www.upsolver.com/) takes the same lake and puts SQL pipelines on it:

> Continuous SQL Pipelines for Cloud Data Lakes. No custom coding. No orchestration. No infrastructure maintenance.

Once data is in the lake or warehouse, the transformations become code. [DBT](https://www.getdbt.com/) is dbt Labs, which empowers data teams to build reliable, governed data pipelines, accelerating analytics and AI initiatives with speed and confidence. The older pitch reads:

> dbt helps data teams work like software engineers—to ship trusted data, faster. collaboratively deploy analytics code following software engineering best practices like modularity, portability, CI/CD, and documentation. Now anyone who knows SQL can build production-grade data pipelines.

The videos that go with dbt run from a first look to production setups. The [intro](https://www.youtube.com/watch?v=R2nr1uZ8ffc) link now opens Streamlit's "1/4: What is Streamlit". The [in depth intro](https://www.youtube.com/watch?v=MHnJDqEKyUY) is AI Council's "Meet dbt: The Data Transformation Tool Used by JetBlue, GitLab, Wistia & Away", from Fishtown Analytics, and [dbt in one hour](https://www.youtube.com/watch?v=na6eu9WSXGY) is dbt Labs' dbt How-to Hour. Moving to deployment, [CI/CD with dbt](https://www.youtube.com/watch?v=snp2hxxWgqk) is HASHMAP on automation and continuous delivery with Snowflake using dbt. [snowflake terraform and dbt](https://www.youtube.com/watch?v=r4tAyiTgwRw) is dbt Labs' talk with Ritual on managing Snowflake architecture with Terraform and dbt, and [hubspot snowflake and dbt](https://www.youtube.com/watch?v=qDHgknWW_oo) is dbt Labs on building a productive data organization at Hubspot with Snowflake and dbt. When the transformation lives in Spark rather than SQL, the tool is [Metorikku](https://github.com/YotpoLtd/metorikku) — a simplified, lightweight ETL framework based on Apache Spark.

Transformed tables are only useful when people can see them, which is the job of BI tools that connect directly to a database. [redash](https://redash.io/) — connect and query your data sources, build dashboards to visualize data, and share them with your company. [Metabase](https://www.metabase.com/) grounds every answer in your semantic layer, your metrics and business logic, so you can inspect the query behind it. Growth FullStack's Metabase page describes it this way:

> [is an easy-to-use, open source business intelligence tool that lets you analyze data from a variety of data destinations and sources. It also follows a simple and fast setup process. Its data visualization capabilities are exceptional and can be showcased in a user-friendly way, without using SQL. With Metabase, you can easily share live dashboards, automated reports, and questions with the rest of your team.](https://growthfullstack.com/analyse/metabase-bi-tool/)

The third BI option is [Superset](https://superset.apache.org/) — Apache Superset is a modern data exploration and visualization platform.

Between the sources and the warehouse sit the tools that move and generate data without custom code. [Stitch](https://www.stitchdata.com/) — Stitch rapidly moves data from 130+ sources into a data warehouse so you can get to answers faster, no coding required. [SnowPlow](https://snowplowanalytics.com/) — generate [complete, accurate and well-structured event data](http://web.archive.org/web/20220601204852/https://snowplowanalytics.com/web-and-mobile-data/) across all platforms and channels in a common format, with the Snowplow Behavioral Data Platform. The archived page makes the case for richness, structure defined before collection, and accuracy, and the live site now calls Snowplow a customer context layer. [Workato](https://www.workato.com/) — a single platform for integration and workflow automation across your organization.

The last tool on the shelf checks that what arrived is right. [AWS Deequ](https://aws.amazon.com/blogs/big-data/test-data-quality-at-scale-with-deequ/) is the AWS Big Data Blog post on testing data quality at scale with Deequ.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Generate complete, accurate and well-structured event data. This address no longer opens: <https://snowplowanalytics.com/web-and-mobile-data/>


# Databases

Data has to live somewhere queryable, and the choice is lake, warehouse, table format, or a specialized engine. This page moves from lakes and warehouses, through comparisons and NoSQL versus relational, into table formats and file formats, then Snowflake, ClickHouse, vector databases, feature engineering, and use cases.

## Lakes and warehouses

The first choice is the shape of the store itself, and the warehouse is where most analytics teams start. [What is a DWH? a comprehensive guide](https://www.oracle.com/database/what-is-a-data-warehouse/) is Oracle's "What Is a Data Warehouse?", on the latest in data warehousing and how it can benefit your business. The [DataLakeHouse](https://www.firebolt.io/blog/snowflake-vs-databricks-vs-firebolt) link now opens the Firebolt team's blog of news, updates, and engineering posts. The idea it pointed at is that Delta Lake is a data lake that can store raw unstructured, semi-structured, and structured data. When combined with Delta Engine it becomes a data lakehouse.

Snowflake is the warehouse that keeps coming up on this page, so it is worth defining early. [What is SnowFlake](https://www.stitchdata.com/resources/snowflake/), [2](http://web.archive.org/web/20230401004452/https://www.slalom.com/insights/snowflake-implementation-success) — Snowflake decouples the storage and compute functions, which means organizations that have high storage demands but less need for CPU cycles, or vice versa, don’t have to pay for an integrated bundle that requires them to pay for both. Users can scale up or down as needed and pay for only the resources they use. The first of those is Stitch on Snowflake as built for the cloud from the ground up, and the second is Slalom's four keys to success with Snowflake. To [get started with SF](https://www.phdata.io/blog/getting-started-with-snowflake/), phData's guide explores the Modern Data Stack, its technologies, strategies for migration, and best practices.

A warehouse is often too broad for one team, which is where the data mart comes in. [talend on data marts](https://www.talend.com/resources/what-is-data-mart/) defines a data mart as a structured data repository purpose-built to support the analytical needs of a particular department, line of business, or geographic region. (good) [netsuite on data marts](https://www.netsuite.com/portal/resource/articles/data-warehouse/data-mart.shtml) — the three types above, plus structures (star, snowflake, denormalized) and comparisons. The [basic intro](https://study.com/academy/lesson/what-is-a-data-mart-design-types-example.html) is the study.com lesson.

The opposite of the curated mart is the data lake, and a lake needs its health watched. The guide to [monitoring health status at scale](https://medium.com/data-science/how-to-monitor-data-lake-health-status-at-scale-d0eb058c85aa) using Great Expectations and Spark is by Davide Romano.

The same notes are in [Data & Model Tests](/evals/data-and-model-tests).

## Comparisons

With lake, warehouse, and mart defined, the next question is which one to pick. [Data Lake vs Data Warehouse](https://www.talend.com/resources/data-lake-vs-data-warehouse/) is Qlik's six key differences: a data lake holds structured and unstructured data, while a data warehouse holds highly structured data processed for a defined purpose. The figure below is that comparison.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-bd397a7548054d88998f2620d2741b9a0666212a%2Fimage%20(14).png?alt=media" alt=""><figcaption><p>Data Lake vs Data Warehouse</p></figcaption></figure>

A BlueGranite list of the top 5 differences between DL and DWH used to sit here; that address now points to an unrelated site and is kept at the end of the page. [Amazon on DL vs DWH](https://aws.amazon.com/big-data/datalakes-and-analytics/what-is-a-data-lake/) is AWS on what a data lake is, how and why businesses use one, and how to use it with AWS; the figure below is from that comparison.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-0f3f2fde9fed5e5d25a9b83d19098e4d1c1f4388%2Fimage%20(36).png?alt=media" alt=""><figcaption><p>Amazon on DL vs DWH</p></figcaption></figure>

The lakehouse blurs the line, and the Firebolt comparison argues you will not pick just one. [Snowflake vs Delta Lake vs Fire Bolt](https://www.firebolt.io/blog/snowflake-vs-databricks-vs-firebolt) now lands on the Firebolt team's blog; the passage kept from it reads:

> Databricks Delta Lake and Delta Engine is a lakehouse. You choose it as a data lake, and for data lakehouse-based workloads including ELT for data warehouses, data science and machine learning, even static reporting and dashboards if you don’t mind the performance difference and don’t have a data warehouse.
>
> Most companies still choose a data warehouse like Snowflake, BigQuery, Redshift or Firebolt for general-purpose analytics over a data lakehouse like Delta Lake and Delta Engine because they need performance.
>
> But it doesn’t matter. You need more than one engine. Don’t fight it. You will end up with multiple engines for very good reasons. It’s just a matter of when.

Once the choice is a warehouse, the question narrows to which engine. [Snowflake vs Amazon Redshift](https://www.sphereinc.com/blogs/snowflake-vs-aws-redshift-which-should-you-use-for-your-data-warehouse/) is Sphere Partners' comparison of the two cloud warehouses, their benefits and functions, and the Snowflake AWS marketplace. For Snowflake on its own there is an [Intro and demo](https://www.youtube.com/watch?v=dUL8GO4ZK9s) video, and [The three pillars - snowflake](https://medium.com/data-science/why-you-need-to-know-snowflake-as-a-data-scientist-d4e5a87c2f3d) is "Why You Need To Know Snowflake As A Data Scientist", written for readers who may be hearing of the company for the first time. [Snowflake vs redshift on medium](https://medium.com/data-science/redshift-or-snowflake-e0e3ea427dbc) is "Redshift or Snowflake!". [SF vs RS](https://www.xplenty.com/blog/redshift-vs-snowflake/) is Mark Smallcombe on the critical differences between Amazon Redshift and the Snowflake data warehouse. Donal Tobin wrote the other two pairings: [RS vs BQ](https://www.xplenty.com/blog/redshift-vs-bigquery-comprehensive-guide/), on choosing the right data warehouse as a critical component of your general data and analytic business needs, and [SF vs BQ](https://www.xplenty.com/blog/snowflake-vs-bigquery/), Snowflake vs. BigQuery.

## NoSQL vs relational

Lakes and warehouses are for analytics; the application behind them still has to choose between a document store and a relational database. The same notes are in [Database Modeling](/data/engineering-intro/database-architecture-and-modeling).

The question is kept as an interview question:

Explain the difference and the reason to choose using NoSQL {mongoDB | DynamoDB | .. } over Relational database {Postgress |MySQL} and vice versa. Give an example for a project where you had to make this choice, and walk through your reasoning.

(This question can be modified for the relevant technologies.)

## Data lake table formats

A lake of raw files only behaves like a database when a table format sits on top of it. The same notes are in [File formats](#file-formats) and [Tools](/data/engineering-intro/tools).

### Apache Iceberg

Iceberg is the table format with the most notes here, from first introduction to production cost. [A short Intro](https://medium.com/expedia-group-tech/a-short-introduction-to-apache-iceberg-d34f628b6799) is Christine Mathiesen's short introduction to Apache Iceberg, and [A Primer](https://thedatafreak.medium.com/apache-iceberg-a-primer-75a63470bfa2) is Ani's primer.

Performance is where the formats get compared. The earlier post, [delta vs iceberg](https://databeans-blogs.medium.com/delta-vs-iceberg-performance-as-a-decisive-criteria-add7bcdde03d), treats performance as the deciding criterion and describes a data lakehouse as an open architecture that brings the scalability and cost-effectiveness of data lakes together with the reliability and performance of data warehouses on one platform. After comparing delta vs iceberg in our previous blog, a lot of people asked for benchmarking their latest versions and for Apache Hudi to be thrown into the mix, and that follow-up is [Benchmarking Delta vs Iceberg vs Hudi](https://databeans-blogs.medium.com/delta-vs-iceberg-vs-hudi-reassessing-performance-cb8157005eb0).

Production is the last test. [How we migrated our production data lake to iceberg](https://medium.com/insiderengineering/how-we-migrated-our-production-data-lake-to-apache-iceberg-4d6892eca6e6) is Deniz Parmaks on moving tens of terabytes in Amazon S3 from Apache Hive to Apache Iceberg, and [How we reduced our cost by 90%](https://medium.com/insiderengineering/apache-iceberg-reduced-our-amazon-s3-cost-by-90-997cde5ce931) is the companion post on how Iceberg reduced their Amazon S3 cost by 90%. [Top 5 Features](https://dipankar-tnt.medium.com/apache-iceberg-features-101-331a254a7ada) is Dipankar Mazumdar on why choosing a table format is one of the most critical decisions in a cloud data lake architecture, and Iceberg's top five features.

### Databricks Delta Lake

Delta Lake is the other format in those benchmarks, and its question is how it connects to the rest of the stack. The same notes are in [Databricks](/ai-engineering/experiment-management#databricks).

[integrating delta lake into other platforms](https://www.databricks.com/blog/integrating-delta-lakehouse-other-platforms) is Databricks on integrating Delta Lakehouse with various platforms to unify the data ecosystem and enhance data management and analytics.

## File formats

Under every table format is a file format, usually Parquet, and these are the interview questions about it. The same notes are in [Data lake table formats](#data-lake-table-formats).

1. Can you explain the parquet file format?
2. How is this leveraged by Spark? <https://databricks.com/session/spark-parquet-in-depth> How this leveraged Spark
3. What are the shortcomings of parquet and how is it solved by file formats like hudi, delta, iceberg? <https://lakefs.io/hudi-iceberg-and-delta-lake-data-lake-table-formats-compared/> answers it by comparing hudi, iceberg, and delta.

## Snowflake

After the formats, the page returns to Snowflake, the warehouse that most of the comparisons above measured against. The same notes are in [DBT](/data/engineering-intro/data-quality#dbt) and [Tools](/data/engineering-intro/tools).

[(good) Guides](https://www.snowflake.com/guides/) is Snowflake's AI Data Cloud Fundamentals, the resource for foundational AI, cloud, and data concepts. [getting started with SF tasks](https://medium.com/snowflake/getting-started-with-snowflake-tasks-945ecd54c77b) SQL procedures schedules tree tasks, by Rajiv Gupta. For the bill, [cost](https://www.phdata.io/blog/what-is-the-snowflake-data-cloud/) is Justin Delisi on what the Snowflake Data Cloud is, how much it costs, and how it breaks down data silos with stronger data sharing and data exchange.

## ClickHouse

Snowflake is a managed warehouse; ClickHouse is the open-source engine for when reports have to be real-time. The [open source database for real time apps and analytics](https://clickhouse.com/) is ClickHouse, a fast open-source column-oriented database management system that generates analytical data reports in real-time using SQL queries.

## Vector databases

After the OLAP engines, embeddings need their own kind of store. The same notes are in [RAG](/generative-ai/rag), [Search](/natural-language-processing-nlp/search), and [VECTOR SIMILARITY SEARCH](/deep-learning/representations#vector-similarity-search).

The analyst view is Gartner — [Innovation Insight: Vector Databases](https://www.gartner.com/doc/reprints?id=1-2HBZK5EN\&ct=240418\&st=sb), a Gartner reprint. The open-source option is Chroma — AI native open source embedding database, on [github](https://github.com/chroma-core/chroma), where it now calls itself search infrastructure for AI.

## Feature engineering

With storage named, the warehouse can also do some of the model's work. The same notes are in [FEATURE ENGINEERING](/data/processing-intro/feature-engineering#feature-engineering).

[Feature engineering in snowflake](https://medium.com/data-science/feature-engineering-in-snowflake-1730a1b84e5b) is James Weakley's follow-up to showing that a true cloud data warehouse can handle common machine learning tasks for structured data, like training tree-based models.

## Use cases

Closing the page, one architecture talk puts these systems together. [Hunters on their architecture](https://www.youtube.com/watch?v=S78gCJ3tdc4), Airflow, Snowflake, Snowpipe, Flink, RocksDB, cluster optimization during ingestion, monitoring metrics, cost.

Two notes from earlier sections also had towardsdatascience.com addresses, kept here. Monitoring health status at scale is Davide Romano's guide to building a Data Quality workflow with Great Expectations and Spark: <https://towardsdatascience.com/how-to-monitor-data-lake-health-status-at-scale-d0eb058c85aa>. The three pillars - snowflake is Admond Lee on the only data warehouse built for the cloud: <https://towardsdatascience.com/why-you-need-to-know-snowflake-as-a-data-scientist-d4e5a87c2f3d>

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* 2. This address no longer opens: <https://www.slalom.com/insights/snowflake-implementation-success>
* Snowflake vs redshift on medium. This address no longer opens: <https://towardsdatascience.com/redshift-or-snowflake-e0e3ea427dbc>
* Feature engineering in snowflake. This address no longer opens: <https://towardsdatascience.com/feature-engineering-in-snowflake-1730a1b84e5b>
* Top 5 differences between DL and DWH. This address now points to an unrelated site: <https://www.bluegranite.com/blog/bid/402596/top-five-differences-between-data-lakes-and-data-warehouses>


# Database Modeling

Storage choices and models decide how a warehouse can be queried later. This page starts from the kinds of databases and the warehouse, then moves to Data Vault, data fabric, and virtualization as ways to model and connect data, and ends with tools for deploying models.

The same notes are in [NoSQL vs relational](/data/engineering-intro/lakes-and-warehouses#nosql-vs-relational).

Before modeling anything, it helps to know which kind of database you are modeling for. [Types of DBs](https://simonatta.medium.com/database-types-2dac81461709) is part of an article series on data and information that walks through database types.

## Data Warehouse

The warehouse is the first target most models are built for, so the page starts there. The same notes are in [Data Patterns](/data/engineering-intro/patterns).

The (good) [a guide from strategy to implementation](https://www.analytics8.com/blog/what-is-a-data-warehouse/) is Sean Costello on what a data warehouse is, what your business may be missing without one, and three questions to ask when deciding on one.

## Data Vault Modeling

A warehouse still needs a modeling method, and Data Vault is the one built for history and auditing. Wikipedia defines it this way:

[**Data vault modeling**](https://en.wikipedia.org/wiki/Data_vault_modeling) is a [database](https://en.wikipedia.org/wiki/Database) modeling method that is designed to provide long-term historical storage of [data](https://en.wikipedia.org/wiki/Data) coming in from multiple operational systems. It is also a method of looking at historical data that deals with issues such as auditing, tracing of data, loading speed and resilience to change as well as emphasizing the need to trace where all the data in the database came from. This means that every [row](https://en.wikipedia.org/wiki/Row_\(database\)) in a data vault must be accompanied by record source and load date attributes, enabling an auditor to trace values back to the source. It was developed by [Daniel (Dan) Linstedt](https://en.wikipedia.org/w/index.php?title=Daniel_Linstedt\&action=edit\&redlink=1) in 2000. — Wikipedia

Databricks gives the working shape. It is a design pattern to build a DWH for enterprise analytics. It has hubs (core business concepts), links (relationships between hubs), and satellites that store info about these two. Good for the lakehouse paradigm. The [link has a good image.](https://www.databricks.com/glossary/data-vault) — Databricks

## Data Fabric

A vault models one warehouse; a data fabric is the architecture that ties many stores together. The same notes are in [Data Catalogs](/data/datasets-intro/data-catalogs), [Data Governance](/data/engineering-intro/data-governance), and [Data Mesh](/data/engineering-intro/data-mesh).

The clearest walk through the flow is Preeti Hemant's. [Data Fabric](https://preetihemant.medium.com/modern-data-architecture-models-69e90b725a05) is a data architecture that follows a set of steps that determine its flow. The first step takes data through an integration phase. In the integration phase, data is ingested and then cleaned, transformed and loaded into storage. Then, there is the data quality phase where quality assessment is performed on the stored data. This data is then made available for different use cases through a combination of a data lake and a data warehouse. Typical use cases are BI, analytics and machine learning. Data governance policies are defined for the ingested data and a data catalog is used for discoverability — by Preeti Hemant.

The vendors each draw the same idea as layers. The (short) [Netapp](https://www.netapp.com/data-fabric/what-is-data-fabric/) link now opens NetApp's unified storage page, about storage that flexes as your data needs scale and evolve. (good) [IBM](https://www.ibm.com/topics/data-fabric) — Data Management layer, Data Ingestion Layer, Data Processing, Data Orchestration, Data Discovery, Data Access. (good) [Gartner](https://www.gartner.com/smarterwithgartner/data-fabric-architecture-is-key-to-modernizing-data-management-and-integration) good the pillars data fabric, from Gartner's business insights for executives. [talend](https://www.talend.com/resources/what-is-data-fabric/) defines data fabric as a machine-enabled data integration architecture utilizing metadata assets to unify, integrate, and govern disparate data environments, and covers why you need it and best practices. (good but lengthy) [spiceworks](https://www.spiceworks.com/tech/big-data/articles/what-is-data-fabric/) good but lengthy architectural components best practices.

Two more break the fabric into components you can draw. (good) [k2view](https://www.k2view.com/what-is-data-fabric) — has a great figure of integration, storage, catalog, cleansing and masking, transformation and enrichment, governance, web services. (good) [tibco](https://www.tibco.com/reference-center/what-is-data-fabric) — application and services, dev and integration, security, storage management, transport, endpoints.

Fabric is often confused with mesh, and [mesh vs fabric](https://www.datanami.com/2021/10/25/data-mesh-vs-data-fabric-understanding-the-differences/) is Datanami on understanding the differences.

## Data virtualization

Fabric integrates data by moving and cataloging it; virtualization is the option of querying it where it already lives. The [data virtualization](https://www.ibm.com/analytics/data-virtualization) link now opens IBM's solutions page, which presents enterprise solutions created to address specific business challenges and needs.

## Tools for deploying data models

Closing the page, a model is only done when it runs in production. [tools for deploying data models in prod](https://www.superdatascience.com/podcast/tools-for-deploying-data-models-into-production) is SuperDataScience episode SDS 619, where Jon Krohn speaks with Erik Bernhardsson, who invented Spotify's original music recommendation system, about interviewing data science candidates, deploying a data model into the cloud, and how Spotify went from digital music startup to AI-driven streaming giant.


# Data Engineering

Data engineering is the job that makes data usable, and it is not the same job as writing product software. This page starts with what the role is, then Job versus Service, then the CAP theorem notes beside the CAP interview questions. The same notes are in [Data Engineering Questions & Training](/data/engineering-intro/data-engineering-questions).

The first thing to get right is that the role has its own shape. [Data engineers are not software developers - a comparison](https://betterprogramming.pub/data-engineering-is-not-software-engineering-af81eb8d3949) is the side-by-side of the two jobs. The role also splits further: [Different Types of data engineers](https://medium.com/coriers/different-types-of-data-engineering-teams-6a1056986d3) is Ben Rogojan on how the data engineer role morphed drastically in a decade, from a time when employers expected data scientists to both calculate eigenvectors and write MapReduce jobs for Hadoop. The teams it separates are data infra, data platform, data engineers, data governance, and BI & analytics.

## Job vs Service

Whichever team a data engineer sits on, the first design choice in an ML pipeline is whether a step runs as a job or as a service. The interview question puts it this way:

Explain the difference and the reason to choose using Job over Service and vice versa. Give an example for a project where you had to make this choice, in the context of ML pipelines and walk through your reasoning.

### [Cap theorem](https://medium.com/data-science/cap-theorem-and-distributed-database-management-systems-5c2be977950e)

A service that keeps state across machines runs into the CAP theorem. The heading links Syed Sadat Nazrul's "CAP Theorem and Distributed Database Management Systems", which starts from the shift away from scaling vertically, with more powerful machines, toward expanding horizontally with more machines doing the same task in parallel. The same notes are in [CAP Theorem](#cap-theorem).

Two sources push the theorem further. [Cap](https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/) is Confluent's "Turning the database inside-out with Apache Samza", from the company building the foundational platform for data in motion. [Cap is changing](https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed/) is InfoQ's "CAP Twelve Years Later: How the 'Rules' Have Changed". The figure below is the CAP triangle, copied from the original hosted image.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-b49f7512b15167760c9e28b303e472eb8219924f%2Fgimg-54fdea866648.png?alt=media" alt=""><figcaption><p>Cap theorem</p></figcaption></figure>

## CAP Theorem

With the theorem stated, the interview questions check whether you can apply it. Each of the first six is answered in FullStack.Cafe's "15 CAP Theorem Interview Questions (ANSWERED) For System Design Interview":

1. [What Is CAP Theorem?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
2. [Can you 'get around' or 'beat' the CAP Theorem?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
3. [Name some types of Consistency patterns](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
4. [What Do You Mean By High Availability (HA)?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
5. [What are A and P in CAP and the difference between them?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
6. [What does the CAP Theorem actually say?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
7. How it affects real world application (latency is availability in real world)

[Another great resource for CAP questions](https://github.com/henryr/cap-faq) is henryr's The CAP FAQ on GitHub. The diagrams below illustrate the theorem.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-d96eb2d33dbc24fbadb341d5b41eb6ac7a54fb97%2F3?alt=media" alt=""><figcaption><p>CAP Theorem</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-b4fcf3d4b7f3ac47ec2302f1e55f2dba52b51d20%2F4?alt=media" alt=""><figcaption><p>CAP Theorem</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-0bc713b64e2fec83c3c4823601f4eba78a7269dd%2F5?alt=media" alt=""><figcaption><p>CAP Theorem</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-cc9eeb33d4300efe044285fa5c5a2430c44225b8%2F6?alt=media" alt=""><figcaption><p>CAP Theorem</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-f57ec0d971c08f063090ee816463120a4fa5128f%2F7?alt=media" alt=""><figcaption><p>CAP Theorem</p></figcaption></figure>

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Cap theorem. This address no longer opens: <https://towardsdatascience.com/cap-theorem-and-distributed-database-management-systems-5c2be977950e>


# Data Architecture

A platform needs an architecture before the warehouses and pipelines stack up. The page walks through a few data-platform architectures, from a modular stack to multi-petabyte and evolving platforms, and ends with an introduction to domain-driven design for data.

The architecture question and the ownership question are the same question seen from two sides. The same notes are in [Data Mesh](/data/engineering-intro/data-mesh) and [ML Architecture](/ai-engineering/ml-architecture).

The first architecture is the smallest one that still holds together: [How to build a modular data stack](https://medium.com/the-prefect-blog/how-to-build-a-modular-data-stack-data-platform-with-prefect-dbt-and-snowflake-89f928974e85) is the modular data-platform write-up. Scale changes the picture. Nielsen shows [a multi-petabyte data platform using EMR](https://www.youtube.com/watch?v=9hUTkBsuIZA), the AWS re:Invent 2020 talk on how they built it on Amazon EMR. A platform is also never finished, and [Evolving and scaling the data platform at Yotpo](https://www.dataengineeringpodcast.com/yotpo-data-platform-architecture-episode-285/) is the podcast episode that treats building a data platform as an iterative, evolutionary process that needs collaboration. Once the platform grows past one team, the boundaries have to follow the business, which is where the [Intro to domain-driven design (DDD)](https://medium.com/latentview-data-services/introduction-to-domain-driven-design-part-1-of-2-6630f72f2853) comes in, the first part of a two-part introduction.


# Data Platforms

A data platform is where compute and storage meet so teams can run jobs without reinventing the lake each time. The page starts with Databricks as the platform idea in practice, then moves to the query engines and their costs in Athena, Spark, and BigQuery, compares Glue DataBrew with Data Wrangler, and closes with a GCP disk note. The same notes are in [Data Product](/data/engineering-intro/data-product) and [ML Architecture](/ai-engineering/ml-architecture).

## Databricks

The first platform is Databricks, and the story there runs from the guarantee it makes (ACID), through its courses and core concepts, into the Spark APIs, optimizations, and best practices underneath.

The same notes are in [Databricks](/ai-engineering/experiment-management#databricks).

A lake only behaves like a database if writes are safe, so the first question is what ACID transactions are. [Databricks is ACID](https://databricks.com/glossary/acid-transactions) is the Databricks glossary entry that answers it, and the figure below is the same point.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-fdabcb8d61774485f081f5b11bdb20d5e39d84fc%2Fimage%20(40).png?alt=media" alt=""><figcaption><p>Databricks is ACID</p></figcaption></figure>

With the guarantee in place, the DB Learning Library is where to learn the platform itself. The [Free courses](https://www.databricks.com/training/catalog?costs=free) are the free slice of the Databricks training catalog, expert-led courses in data science, machine learning, and big data analytics. The [Docs Optimization recommendations](https://docs.databricks.com/en/optimizations/index.html) are the documentation on optimizations and performance recommendations on Databricks, and the [Comprehensive Guide to Optimize Databricks, Spark and Delta Lake Workloads](https://www.databricks.com/discover/pages/optimize-data-workloads-guide) collects best practices and strategies for making data workloads faster and more efficient. Two starter courses split the platform by role: DB [for ML](https://www.databricks.com/training/catalog/get-started-with-databricks-for-machine-learning-2460) tours the workspace and notebooks, then exploratory data analysis, feature engineering, and tracking and managing models with MLflow, while [DB for Data Engineering](https://www.databricks.com/training/catalog/get-started-with-databricks-for-data-engineering-1511) covers the foundations of a basic data engineering workflow, the workspace, Unity Catalog, and the day-to-day building blocks, through paired demos and labs.

Outside the official library, the (good) [Introduction & Tutorial](https://medium.com/@chuck.connell.3/databricks-a-history-and-introduction-438ce827227) is a good walk through cluster, notebook, table, SQL, DataFrame, and connections. The [must know 7 concepts](https://www.datacamp.com/tutorial/introduction-to-databricks) are DataCamp's tutorial on the seven must-know Databricks concepts for any data specialist, and the two figures below come from it.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-292ead298d6c9254846528bb0cb9b993d710db4b%2Fimage%20(41).png?alt=media" alt=""><figcaption><p>must know 7 concepts</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-65ea0cb5c35a260601048c203ac222e21ff47fc2%2Fimage%20(42).png?alt=media" alt=""><figcaption><p>must know 7 concepts</p></figcaption></figure>

Under the notebooks sits Spark, and the first API choice is RDD vs Dataframe vs Dataset. The [2016 official blog post](https://www.databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html) is the tale of three Apache Spark APIs: their performance, their optimization benefits, and when to use each. The [linkedin blog post](https://www.linkedin.com/pulse/rdd-vs-dataframe-dataset-sanyam-jain-iwsfe/) goes over the differences among the three again, the [comparison on youtube](https://www.youtube.com/watch?v=aBUqIAGxeg8) is the video version of the same comparison, and [RDDs vs. Dataframes vs. Datasets – What is the Difference and Why Should Data Engineers Care?](https://www.analyticsvidhya.com/blog/2020/11/what-is-the-difference-between-rdds-dataframes-and-datasets/) starts from Apache Spark being the first choice for data engineers and explains why the difference matters to them.

Once the API is chosen, the next concern is optimizations. [Optimization recommendations on Databricks](https://docs.databricks.com/en/optimizations/index.html) is the same documentation page on performance recommendations named above, and the [Comprehensive Guide to Optimize Databricks, Spark and Delta Lake Workloads](https://www.databricks.com/discover/pages/optimize-data-workloads-guide) is the same best-practices guide, here read for tuning rather than learning. [Why and How: Partitioning in Databricks](https://medium.com/@eduard2popa/why-and-how-partitioning-in-databricks-e9e6f960db43) is Eduard Popa on the single most common lever, partitioning.

Best practices close the Databricks story: the [official docs](https://docs.databricks.com/en/delta/best-practices.html) give the best practices and recommendations for using Delta Lake on Databricks.

## Athena

After Databricks, Athena is the serverless query engine where the questions turn to engine, cost, and performance tuning. The list below is a question checklist; the tuning item links the AWS post with its top-10 tips.

1. What is the engine behind Athena
2. How is Presto different from Spark? How does it affect your query planning?
3. [Performance tuning](https://aws.amazon.com/blogs/big-data/top-10-performance-tuning-tips-for-amazon-athena/) — Top 10: partitioning, bucketing, compression, optimize file sizes, optimize columnar data store generation, query tuning, optimize order by, optimize group by, use approx functions, column selection. What are the tradeoffs (time vs cost)?
4. What is the cost composed of?
5. How can you calculate cost?
6. How can you optimize your queries (partitions, join order, limit tricks, etc)
7. What options do you have to limit the cost of Athena?
8. When would you use Athena vs Spark

## Spark

The last Athena question, when to use Athena vs Spark, leads straight into Spark itself: join strategies, DataFrame vs Dataset, and adaptive query execution.

The same notes are in [Spark](/ai-engineering/experiment-management#spark).

The articles by sivaprasad mandapati are a good pool here: [Several spark articles that can be used as candidate questions](https://medium.com/@sivaprasad-mandapati) is his Medium page, and several of the notes below come from it.

Joins are where most Spark jobs spend their time. [Join strategies #1](https://medium.com/datakaresolutions/optimize-spark-sql-joins-c81b4e3ed7da) is Prabhath Vemula's best practices and optimization techniques from joining two big tables in Spark SQL, and [Join strategies #2](https://medium.com/data-science/strategies-of-spark-join-c0e7b4572bcf) walks the join strategies Spark uses internally, which helps with tricky joins and out-of-memory root causes. The questions to ask of both are how? Pros and cons. (broadcast hash, shuffle hash, shuffle sort merge, cartesian). A related question is what’s the difference between a data frame and a dataset?

The choice between the two most common strategies is [Sort merge vs broadcast](https://medium.com/swlh/spark-joins-tuning-part-1-sort-merge-vs-broadcast-a98d82610cf0), a tuning article that starts from partitions spread across executors as Spark's bread and butter. Its finding is that a broadcast join is 4 times faster if one of the table is small and enough to fit in memory. Is broadcasting always a good solution? Absolutely no. If you are joining two data sets both are very large broadcasting any table would kill your spark cluster and fails your job.

When neither side is small, the lever is the shuffle, which is the subject of [Shuffle & AQE](https://medium.com/@sivaprasad-mandapati/spark-joins-tuning-part-2-shuffle-partitions-aqe-8688cb23317b), the second part of that tuning series. Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan. It does three things:

* Dynamically coalescing shuffle partitions
* Dynamically switching join strategies
* Dynamically optimizing skew joins

## BigQuery

After Spark, BigQuery brings back the Athena kind of question on another cloud: cost, partitions, clustering, and access. The checklist is:

1. What is the difference in the implementation between partitions and clustering in BQ?
2. What ways do you know to reduce query cost in BigQuery?
3. What is the BigQuery cost composed of? How can you reduce storage cost?
4. Did you ever encounter a memory error when running BigQuery? Why does it happen and how is it related to the Dremel implementations
5. How can you control the access to sensitive data in BigQuery?
6. What options do you have to limit the cost of BigQuery?
7. When using BigQuery ML to train TF models — what happens in the background?

## AWS Glue DataBrew vs Data Wrangler

Beside the query engines, data still has to be prepared before training, and AWS offers two tools for it. [Julien simon on AWS glue data brew vs data wrangler](https://julsimon.medium.com/data-preparation-aws-glue-data-brew-or-amazon-sagemaker-data-wrangler-d8e76d1510cb) asks which one to use for data preparation, AWS Glue DataBrew or Amazon SageMaker Data Wrangler.

## GCP

Closing the cloud shelf is a practical GCP problem: running out of disk mid-download or mid-training. [Resize google disk size](https://medium.com/google-cloud/resize-your-persist-disk-on-google-cloud-on-the-fly-b3491277b718) shows how to resize a persistent disk on the fly instead of stopping the VM and attaching a new disk. The references [1,](https://cloud.google.com/compute/docs/disks/add-persistent-disk) and [2](https://www.cloudbooklet.com/how-to-resize-disk-of-a-vm-instance-in-google-cloud/) back it up: the first is the Google Cloud documentation on creating and attaching persistent disk volumes on Windows and Linux VMs, and the second resizes a Compute Engine persistent disk without downtime or restarting the instance.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Vector Search. This address no longer opens: <https://www.databricks.com/training/catalog/new-capability-overview-vector-search-2535>
* How I Use Caching in Databricks to Increase Performance and Save Costs. This address no longer opens: <https://blog.det.life/caching-in-databricks-explained-68c07bf1f76b>


# Data Patterns

A row that changes over time needs a rule for what the history looks like. The pattern for that is slowly changing dimensions: what to do when a row changes, from keeping nothing to keeping every version.

The rule belongs to the warehouse design, not to one pipeline. The same notes are in [Data Warehouse](/data/engineering-intro/database-architecture-and-modeling#data-warehouse).

The reference is Adatis. [Slowly changing dimensions (SCD)](https://adatis.co.uk/introduction-to-slowly-changing-dimensions-scd-types/) by Adatis: what you can do when the information in your table changes, types 0 through 6. The figure below lays those types out side by side, with the credit to Adatis.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-a32a74fa34ef09d8a58e40fdbdc9c2c19d5b8e44%2Fimage%20(12).png?alt=media" alt=""><figcaption><p>Slowly changing dimensions</p></figcaption></figure>


# Data Quality

Bad rows break models and dashboards, so quality has to be defined and tested, not assumed. The page starts with what quality means and how to measure it, then turns to dbt as the tool that tests it, and ends with data validation notes and tools. The same notes are in [Data Contract](/data/engineering-intro/data-contract), [Data Lineage](/data/engineering-intro/data-lineage), [Data Observability](/data/engineering-intro/data-observability), [Data Testing](/evals/data-and-model-tests#data-testing), and [Data Validation](#data-validation).

The first point is that quality is not a tool you buy. [why cant data quality be fixed with tech](https://www.analytics8.com/blog/why-data-quality-cannot-be-fixed-with-technology/) is Sharon Rehana on why data quality matters, where bad data comes from, and how to fix those issues. Once the problem is owned, it has to be defined: [CloverDX - data quality](https://www.cloverdx.com/explore/data-quality) explains what data quality really means, how to measure it, and the common best practices and pitfalls.

Measuring means picking metrics. [CloverDX - 6 Data Quality Metrics You Can't Afford To Ignore](https://www.cloverdx.com/blog/6-data-quality-metrics-you-cant-ignore) names six core data quality metrics, how they differ from dimensions and KPIs, and how to track them for reliable AI outcomes. [FirstEigan - data quality metrics](https://firsteigen.com/blog/6-key-data-quality-metrics-you-should-be-tracking/) is Seth Rao's version of six key metrics, with examples and ways to automate them on Azure and Snowflake. [Dataladder - data quality metrics](https://dataladder.com/10-data-quality-metrics-you-should-measure/) widens the list to ten metrics you should measure. [Datacademia - data quality metrics](https://datacadamia.com/data/quality/metric) adds a warning about naming, that these metrics are often called dimensions although in a dimensional model they act as attributes of the data rules, and defines timeliness as the time between when information is expected and when it is readily available. At big-data scale quality becomes veracity, one of the V's: [Open sistemas - big data challenges - veracity](https://opensistemas.com/en/the-four-vs-of-big-data/) is the OpenSistemas overview of the Four V's of Big Data.

## DBT

Once the metrics are named, they need to run as tests on every build, and dbt is where that happens in the warehouse.

The same notes are in [Snowflake](/data/engineering-intro/lakes-and-warehouses#snowflake) and [Tools](/data/engineering-intro/tools).

The starting point is the [DBT builtin tests](https://docs.getdbt.com/docs/building-a-dbt-project/tests), the dbt docs on configuring data tests to assess the quality of input data and the accuracy of the resulting datasets. [DBT expectations](https://hub.getdbt.com/calogica/dbt_expectations/0.1.2/) is the dbt package-hub entry that extends those tests. When neither fits, [DBT custom generic tests](https://docs.getdbt.com/guides/legacy/writing-custom-generic-tests) shows how to define your own custom generic data tests. Those tests are written in Jinja, and [Jinja](https://www.youtube.com/watch?v=OraYXEr0Irg) is Real Python's introduction to Jinja templating.

## Data Validation

dbt tests the warehouse; validation is the wider habit of guarding every place bad data can enter, and these notes come from the question bank.

The same notes are in [Data Quality](/data/engineering-intro/data-quality), [Data Testing](/evals/data-and-model-tests#data-testing), and [Tools](/data/engineering-intro/tools).

The question is: how can you protect yourself from bad data? Data validation, TDDA, monitoring. Each answer has tools. Type validation: [typeguard](https://github.com/agronholm/typeguard) is a run-time type checker for Python. Data validation: [pydantic](https://pydantic-docs.helpmanual.io/usage/dataclasses/) validates Python dataclasses. Test driven: [tdda](https://github.com/tdda/tdda) is the test-driven data analysis functions library. Data quality: [great expectations](https://greatexpectations.io/) is GX Core, an open source framework for testing, validating, and documenting data quality across pipelines, workflows, and teams. Saas: [SuperConductive by GE](https://superconductive.ai/) is the hosted side of the same Great Expectations platform.


# Data Observability

Pipelines fail quietly unless someone watches freshness, volume, and schema drift. Observability is that watching, done continuously instead of after a dashboard breaks. The page moves from the idea to the tools that do the watching, and then to what to do when an outage happens anyway. The same notes are in [Data Quality](/data/engineering-intro/data-quality).

## Tools

Watching data health by hand does not scale, so the idea turns into a set of tools and a cycle for acting on what they find.

[Monte Carlo](https://www.montecarlodata.com/) pitches itself as going beyond data quality to end-to-end data and AI observability for enterprise teams. [BigEye](https://www.bigeye.com/) is the data and AI trust platform for large enterprises, combining data observability, end-to-end lineage, and governance. [Databand](http://web.archive.org/web/20231208161414/https://databand.ai/), read through its archived page, is the proactive observability platform that tries to catch bad data before it impacts the business. Its explainer, [What is data observability](http://web.archive.org/web/20231208161417/https://databand.ai/data-observability/), defines the term as an umbrella of activities and technologies that let you identify, troubleshoot, and resolve data issues in near real-time, and it frames the work as a cycle:

> The DataOps cycle outlines the fundamental activities needed to improve how data is managed within the DataOps workflow. This cycle consists of three distinct stages: Detection, Awareness, and Iteration.

The figure shows those three stages as a loop.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-c050763b1a079d312549a8ddcb0182fcf5d7bd95%2Fimage%20(30).png?alt=media" alt=""><figcaption><p>The DataOps cycle</p></figcaption></figure>

For an open-source option, [DQOps](https://dqops.com) is a data quality platform for the whole data platform lifecycle, from profiling new data sources to automating data quality monitoring.

## Outage handling

Detection only helps if the pipeline can recover once something breaks, and the architecture decides how painful that recovery is.

The note is about outage handling and the differences between stream-based processing vs concurrent isolated worker-based processing using

Q: you have a real time stream — what is better? A stream-based processing system, or a worker-based, that can be triggered on different time ranges, in the context of recovery from outage.

The figure below sketches that comparison; the credit is nielsen Ilai Malka.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-65b42755f689ac98a80ea332e73d2c9ff1dc4a74%2F8?alt=media" alt=""><figcaption><p>Outage handling</p></figcaption></figure>

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Databand. This address no longer opens: <https://databand.ai>
* What is data observability and what is the dataOps cycle. This address no longer opens: <https://databand.ai/data-observability/>


# Data Lineage

When a number is wrong, someone has to walk the path the data took. The page defines data lineage and why it matters, then points at a few good articles, and ends with the vendors that build lineage tools.

Lineage is where governance and quality meet: you cannot own or trust a number you cannot trace. The same notes are in [Data Governance](/data/engineering-intro/data-governance) and [Data Quality](/data/engineering-intro/data-quality).

Data lineage refers to the detailed history of data as it moves through various stages and transformations in an information system. It is essentially the life cycle of data, from its origins to its endpoint, including how it is modified and processed over time. Understanding data lineage is crucial for several reasons:

* Traceability — it helps track where data comes from, which is vital for debugging issues, understanding dependencies, and ensuring data quality.
* Compliance — many regulatory requirements, such as GDPR and HIPAA, require knowing the flow of data to ensure it is handled securely and within legal parameters.
* Data Governance — it aids in managing data, understanding its utility, and ensuring that data usage is consistent with organizational policies.
* Impact Analysis — it allows organizations to assess the potential impact of changes in the data environment. This is crucial for risk management and strategic planning.
* Audit and Reporting — data lineage provides transparency for audits, ensuring that all data used in financial reporting, for instance, is accurate and verifiable.

Those reasons all depend on the trail actually existing. Tools and systems that manage data lineage collect metadata from various parts of data handling systems, providing a visual or documented trail of how data flows through software and systems, which transformations it undergoes, and how it is used in different analyses and decisions. This capability is particularly important in complex systems where data is handled across various platforms and services.

## Good articles

The definition above is the short version; these three articles are the longer reads. The first two answer what data lineage is, from Octopai and Ardoq, and the third is Select Star's complete guide to its benefits, techniques, and best practices.

## Data lineage vendors

Once the trail is understood, someone has to collect it, and that is what the lineage vendors sell. Many of them are also data catalogs.

The same notes are in [Data Catalogs](/data/datasets-intro/data-catalogs).

The cards below are Octopai, Collibra, Azure Purview, Cloudera, Alation, and the open-source Apache Atlas. Alation now presents its product as the Alation Intelligence Operating System, aimed at getting enterprise AI right and keeping it right.


# Data Contract

Producers and consumers need a promise about the shape and meaning of a table. The page first defines data contracts and collects the articles that argue for them, then turns to the implementation options, from JSON Schema to protobuf and a schema registry.

A contract is what turns a table into a product and what makes quality testable. The same notes are in [Data Product](/data/engineering-intro/data-product) and [Data Quality](/data/engineering-intro/data-quality).

The definition comes from service design. A data contract is **a formal agreement between a service and a client that abstractly describes the data to be exchanged**. That is, to communicate, the client and the service do not have to share the same types, only the same data contracts. — [Microsoft](https://learn.microsoft.com/en-us/dotnet/framework/wcf/feature-details/using-data-contracts)

The argument for bringing that idea into data comes from a series by Chad Sanderson. The following is a set of articles around the topics of data product / contract etc. however the focus is about data contracts and expectations IMO, therefore I place it here. — by Chad Sanderson. It starts with the problem: [the existential threat of data quality](https://dataproducts.substack.com/p/the-existential-threat-of-data-quality), and why the Modern Data Stack can't solve it. Then the missing discipline: [the death of data modeling part 1](https://dataproducts.substack.com/p/the-death-of-data-modeling-pt-1), and why it must be resurrected in the Modern Data Stack. Then the people problem: [data's collaboration problem](https://dataproducts.substack.com/p/datas-collaboration-problem), and why the Modern Data Stack can't scale without fixing it. The answer is the rise of [data contracts](https://dataproducts.substack.com/p/the-rise-of-data-contracts), and why data pipelines don't scale without them. [production grade data products](https://dataproducts.substack.com/p/the-production-grade-data-pipeline) is the production-grade data pipeline, and why tech debt accumulates without contracts. And (finally!) [a guide to data contracts p1](https://dataproducts.substack.com/p/an-engineers-guide-to-data-contracts) is the engineer's guide, implementing data contracts for entities.

Beyond that series, the (good) [contracts, robustness in datamesh (but not only)](https://medium.com/data-science/data-contracts-ensure-robustness-in-your-data-mesh-architecture-69a3c38f07db) is by Piethein Strengholt — also has about data-sharing agreements. The (good) [from zero to hero](https://medium.com/data-science/data-contracts-from-zero-to-hero-343717ac4d5e) is good mehdio, pragmatic tips on how to get started with data contracts after all the social-media noise about what they are. [data contract experimentation](https://www.newsletter.swirlai.com/p/sai-04-data-contracts-experimentation) is a SwirlAI newsletter issue covering data contracts alongside Spark shuffle, experimentation environments, and real-time model observability.

## Implementation

A promise needs a format both sides can check, so the next step is choosing how to write the contract down.

The simplest format is [**JSON Schema**](https://json-schema.org/), a vocabulary that allows you to **annotate** and **validate** JSON documents; the [example](https://json-schema.org/learn/miscellaneous-examples.html) page shows it in practice.

The heavier option is protobuf & gRPC. The source is [Protobuf on git](https://github.com/protocolbuffers/protobuf/tree/main/python), the Python part of the repo for Google's data interchange format. [google dev](https://developers.google.com/protocol-buffers) is the Protocol Buffers home page, describing them as language-neutral, platform-neutral extensible mechanisms for serializing structured data. [what is gRPC?](https://grpc.io/docs/what-is-grpc/introduction/) is the introduction to gRPC and protocol buffers, and [what are proto buffers, what do they solve and what are the benefits?](https://developers.google.com/protocol-buffers/docs/overview) is the protobuf overview that answers those questions. For hands-on work, gRPC — [a basic pythonic tutorial](https://grpc.io/docs/languages/python/basics/) is the basic introduction to gRPC in Python, and Protobuf — [a pythonic tutorial](https://developers.google.com/protocol-buffers/docs/pythontutorial) is the Python programmer's introduction to working with protocol buffers.

Community write-ups fill in the why. (good) Intro to gRPC & Protobuf by. [Trevor Kendrick](https://medium.com/@trevor.kendrick?source=post_page-----c21054ef579c--------------------------------) links his Medium page. (good) What are Protocol Buffers and why they are widely used? by [Dineshchandgr](https://medium.com/@dineshchandgr?source=post_page-----cbcb04d378b6--------------------------------) is the second. [Understanding protobufs](https://medium.com/danielpadua/understanding-protocol-buffers-protobuf-a466d8943df8) is the third, by [Daniel Padua Ferreira](https://medium.com/@danielpadua?source=post_page-----a466d8943df8--------------------------------), and it opens with this summary:

> Protocol Buffers (protobuf) is a method of serializing structured data which is particulary useful to communication between services or storing data. It was designed by Google early 2001 (but only publicly released in 2008) to be smaller and faster than XML. Protobuf messages are serialized into a [binary wire](https://developers.google.com/protocol-buffers/docs/encoding) format which is very compact and boosts performance.

One more explainer is protobuf what and why? by [Swaminathan Muthuveerappan](https://medium.com/@swamim?source=post_page-----fcb324a64564--------------------------------). Slightly off topic — [how to choose between grpc, graphql, rest](https://ashish-bania.medium.com/the-exhaustive-guide-to-choosing-between-grpc-graphql-and-rest-b7e4fd6d547e) is the exhaustive guide to picking the right API technology among the three, a choice that shapes the technology roadmap for years.

Once there are many contracts, they need a home. Managing proto files and other schema types such as avro or json schema can be done in [kafka's schema registry](https://docs.confluent.io/platform/current/schema-registry/index.html).

The same notes are in [Kafka](/ai-engineering/full-stack-and-ops-1/key-value-db#kafka).

The two (good) articles from above also exist at their original addresses. (good) contracts, robustness in datamesh (but not only) is at <https://towardsdatascience.com/data-contracts-ensure-robustness-in-your-data-mesh-architecture-69a3c38f07db>. Data Contracts - From Zero To Hero, (good) from zero to hero, a pragmatic approach to data contracts, is at <https://towardsdatascience.com/data-contracts-from-zero-to-hero-343717ac4d5e>.


# Data Governance

Someone has to own who may change the data and what “good” means across teams. The page starts with definitions of data governance, then moves to guides for building it in Snowflake and a thorough survey article, and ends with concrete action items and starting steps.

Governance decides the rules that the fabric enforces and the lineage records. The same notes are in [Data Fabric](/data/engineering-intro/database-architecture-and-modeling#data-fabric) and [Data Lineage](/data/engineering-intro/data-lineage).

The first definition is the [Definition](https://www.cleverrepublic.com/what-is/data-governance/) page from Clever Republic, a consultancy that implements data governance, with the definition taken from DAMA. The DAMA image it once pointed to no longer opens and is kept at the end of the page. [What is](https://www.imperva.com/learn/data-security/data-governance/) is Imperva's shorter version: data governance is a set of practices and processes for managing important data across an organization, with frameworks and tools to help.

From definition to build: How to build [part 1](https://www.phdata.io/blog/snowflake-data-governance/) is Nick Goble on why data governance controls are critical for consistent, dependable data and how to set them up in Snowflake. [part 2](https://www.phdata.io/blog/snowflake-data-governance-2/) continues with access controls, arguing that Snowflake needs a defined information architecture, a process for providing access, and a group responsible for managing those rules and patterns. For the whole field in one place, [a very thorough article](https://bi-survey.com/data-governance/) is the BARC survey piece, and the figure below comes from it.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-981d362ddb22cc99956f702d484b6fff1edf5915%2Fimage%20(4).png?alt=media" alt=""><figcaption><p>a very thorough article</p></figcaption></figure>

Reading is not starting, so the page ends with what to do first. The [action items](https://data.wisc.edu/data-governance/) are the data governance page of UW–Madison's Data, Academic Planning & Institutional Research office. [8 steps to start your data governance program](https://www.analytics8.com/blog/8-steps-to-start-your-data-governance-program/) is Jenna O'Jea's step-by-step guide to a data governance strategy fitted to the organization, with its components and benefits.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* DAMA. This address no longer opens: <https://www.dama.org/cpages/dmbok-2-image-download>


# Data Mesh

A central data team can become the bottleneck, so mesh treats domains as owners of data products. The page goes from what a data mesh is, through Zhamak Dehghani's own introductions and principles, to comparisons with lakes and fabric, topologies, and the practice notes of vendors and teams who built one.

Mesh is an architecture choice and a sibling of fabric, so it is worth reading with both. The same notes are in [Data Architecture](/data/engineering-intro/data-architecture) and [Data Fabric](/data/engineering-intro/database-architecture-and-modeling#data-fabric).

The first question is what a [data mesh](https://databricks.com/session_na20/data-mesh-in-practice-how-europes-leading-online-platform-for-fashion-goes-beyond-the-data-lake) is, and in practice: the Databricks session on how Europe's leading online fashion platform went beyond the data lake.

The idea is Zhamak Dehghani's, so her own material comes next. [Introduction to Data Mesh](https://www.youtube.com/watch?v=_bmYXWCxF_Q) is her Thoughtworks introduction. [how to move from data lake to distributed data mesh](https://martinfowler.com/articles/data-monolith-to-mesh.html) is the article that names the problems with the centralized data lake and says a data mesh needs domains, self-service platforms, and product thinking. [principles and logical architecture](https://martinfowler.com/articles/data-mesh-principles.html) is the follow-up with the four principles that drive a mesh's logical architecture. The (good) [Keynote - Data Mesh by Zhamak Dehghani](https://www.youtube.com/watch?v=L_-fHo0ZkAo) makes the core claim that the pipeline of oltp→etl→olap is broken. [lessons from the trenches](https://www.youtube.com/watch?v=Nw_bxIyR1L0) is Zhamak Dehghani and Sina Jahan on what they learned implementing it.

With the principles in hand, the next step is placing mesh next to its neighbors. [lake vs mesh, he probably means fabric vs mesh](https://medium.com/codex/data-lakehouse-vs-data-mesh-bfa1132f94b) is the data lakehouse vs data mesh comparison, read with that correction in mind. [data mesh 101](https://www.youtube.com/watch?v=hgKOpEQaqdY\&list=PLa7VYi0yPIH0L8ahQYbyBFkGc6a949-Lj\&index=10) — by Confluent is the hands-on episode on creating a data product. The (good) [mesh topologies](https://medium.com/data-science/data-mesh-topologies-and-domain-granularity-65290a4ebb90) — by Piethein Strengholt covers topologies and domain granularity, drawn from his customer conversations and architecture sessions at Microsoft where mesh kept coming up. The figure below is from it.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-e9ad0df8e95b29dbd58a59528ac4ae9fd9e526b8%2Fimage%20(37).png?alt=media" alt=""><figcaption><p>mesh topologies</p></figcaption></figure>

The remaining sources are practice. [Data Mesh & MLOps using a data platform.](https://medium.com/swlh/building-a-data-platform-to-enable-analytics-and-ai-driven-innovation-1bd95e37efb9) ties the mesh to MLOps through a data platform. [Explained](https://medium.com/@david.c.dupuis/data-mesh-explained-a95b6ae50878) is an in-depth explainer written by someone learning data mesh from scratch and sharing it. [The next gen of architecture](https://datagrad.medium.com/data-mesh-transition-to-next-generation-of-data-architecture-832c4bc27e9f) argues that architecture defines the efficiency, scalability, and usability of data, and presents mesh as the transition to the next generation. [how not to mesh it - monte carlo](https://www.montecarlodata.com/blog-what-is-a-data-mesh-and-how-not-to-mesh-it-up/) is Monte Carlo's beginner's guide to implementing a mesh without messing it up. [building data mesh using lake house approach](https://www.youtube.com/watch?v=YPYODx4Pfdc) by AWS is the tech talk on building a mesh architecture with AWS Lake Formation. The [enterprise data mesh by oracle](https://www.oracle.com/a/ocom/docs/datamesh-ebook.pdf) is Oracle's ebook on the enterprise version. [deconstructing data mesh principles](https://medium.com/slalom-data-ai/data-mesh-232e50f42e66) closes by taking the principles apart again.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* (good) mesh topologies - by Piethein Strengholt. This address no longer opens: <https://towardsdatascience.com/data-mesh-topologies-and-domain-granularity-65290a4ebb90>


# Data Product

A platform that nobody adopts is not a product; the work is to treat data as something with users and owners. The page starts with a definition checklist and a framework for a data product, then moves to treating the data platform itself as a product.

A product needs a contract with its consumers and a platform to run on. The same notes are in [Data Contract](/data/engineering-intro/data-contract) and [Data Platforms](/data/engineering-intro/data-platforms).

The definition comes as a checklist: the [Definition, phdata.io](https://offers.phdata.io/hubfs/Checklists/Data%20Product%20Definition%20Checklist_D2.pdf) starts by asking whether there is a general description of what you want to do with the data and what the user is looking for. A checklist says what to ask; a framework says who answers. The (good) [A framework for a data product](https://www.datasciencecentral.com/data-product-framework/) is good on accountability, boundaries, contracts, expectations, consumers, and knowledge.

The same thinking applies one level up, to the platform that serves those products. [A meta discussion on a data platform as a product](https://medium.com/stuart-engineering/how-were-building-our-data-platform-as-a-product-f89142b6547f) is one team's account of building their data platform as a product. [Build a data platform as a product](http://web.archive.org/web/20220707112131/https://productcoalition.com/how-to-design-and-build-a-data-platform-as-a-product-d22329ff5a3f), read through its archived copy, is Filipe Rigueiro's breakdown of the challenges, teachable moments, frameworks, and recommendations for delivering a data platform successfully.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* build a data platform as a product. This address no longer opens: <https://productcoalition.com/how-to-design-and-build-a-data-platform-as-a-product-d22329ff5a3f>


# Data Strategy

A company needs a strategy for what data work is for before it buys another tool. The page starts with the best places to get started, moves through step-by-step guides and frameworks, then the definitions and the reasons strategies fail, and ends with longer series and specific lenses such as ML, product, and competitive advantage.

The (excellent) [how to get started](https://medium.com/data-science/data-strategy-716059d5c066) is the first read. Analytics 8 (excellent) is the second stop: [7 elements of data strategy](https://www.analytics8.com/blog/7-elements-of-a-data-strategy/) is Christina Salmi on the key elements of a data strategy framework that lets decisions rest on analysis rather than intuition, and you can deep dive into other articles from there. [7 tips for strong data infra](https://www.analytics8.com/blog/7-tips-for-strong-data-infrastructure/) covers the clean data and flexible infrastructure that analytics needs to drive decisions, and [step 2/5 how to develop it](https://www.analytics8.com/blog/how-to-develop-data-strategy-roadmap/) is the roadmap step, built to earn executive trust, drive adoption across departments, and prove ROI.

Not every organization is a business. The (very good) [research data strategy - for research orgs](https://medium.com/data-science/research-data-strategy-2fd447f83719) starts from science having entered its big data era, in fields such as bioinformatics, astronomy, particle physics, and geology, while much of the data innovation happened in the tech sector.

Once the goal is clear, the questions have to be asked in order. [how to create an effective DS](https://medium.com/method-perspectives/how-to-create-an-effective-data-strategy-ffedb1bdc71a) — helps to ask the right questions on several topics, starting from the point that gathering and storing data gives no advantage without a plan and a goal. A strategy also needs owners, which is why the (needed) [RACI matrix](https://en.wikipedia.org/wiki/Responsibility_assignment_matrix) is here, the Wikipedia page on the responsibility assignment matrix. [How to build an actionable data strategy phdata.io](https://www.phdata.io/blog/how-to-build-an-actionable-data-strategy-framework/) is Shane Robins's guide to a more practical and focused data strategy framework.

The guides come next. A two-part article starts with [how to create a data strategy for your org](https://medium.com/data-science/how-to-create-a-data-strategy-for-your-organization-e0493110b2e7) and continues with [how to create a perfect data strategy](https://medium.com/data-science/how-to-create-a-perfect-data-strategy-7e8fd9bbfad0). [a step by step guide](https://medium.com/data-science/data-strategy-a-step-by-step-guide-91529a75c72b) is Arthur Mello on how data work fits the company's overall picture and how to draw a roadmap for getting value out of data.

Guides assume the strategy will work; the next sources look at why it often does not. [why is your data strategy failing](https://medium.com/geekculture/why-your-data-strategy-is-failing-19e569b6b082) is Lewis Gavin on the failure side. [Become Data-Driven or Perish: Why your company needs a Data Strategy and not just more Data People](https://medium.com/data-science/become-data-driven-or-perish-why-your-company-needs-a-data-strategy-and-not-just-more-data-people-aa5d435c2f9) is Dwayne Gefferie's argument, drawn from fourteen years of working with data. [create a data strategy](https://medium.com/data-science/data-strategy-make-data-count-fa42b42d409) is a meta discussion, Adam Votava asking whether businesses really use data to decide better, refine operations, and create revenue. [data strategy for AI](https://medium.com/mmc-writes/the-ai-playbook-how-to-develop-a-data-strategy-for-ai-d74df9486c0e) is David Kelnar's blueprint, since executing an AI strategy needs an effective data strategy. [how to create a successful data strategy](https://medium.com/data-science/how-to-create-a-successful-data-strategy-1293bacf463c) is one more guide on getting it right.

The vocabulary matters as well. Data Strategy describes a “set of choices and decisions that together, chart a high-level course of action to achieve high-level goals.” This includes business plans to use information to a competitive advantage and support enterprise goals. That is the [Definition taken from DAMA](http://web.archive.org/web/20221208205957/https://www.cleverrepublic.com/what-is/data-strategy/), read through the archived CleverRepublic page. The same site also defined what is [data management](http://web.archive.org/web/20221208204222/https://www.cleverrepublic.com/what-is/data-management/) and [intelligence](https://www.cleverrepublic.com/what-is/data-intelligence/); the second address now opens a junior data intelligence consultant posting about helping organisations get the maximum value from their data.

Strategy then comes down to which data is worth having. [Data Strategy: Good Data vs. Bad Data](https://medium.com/data-science/data-strategy-good-data-vs-bad-data-d40f85d7ba4e) tells the story of Signet Bank in 1990, which turned its customer credit department into a laboratory that tested credit terms and collected data for years. [Data strategy in progress](https://dasbarrett.medium.com/data-strategy-in-progress-53e5e341ff7) is Dan Barrett on the data strategy work at Citizens Advice, a strategy seen while it is still being built.

For a longer read, Francesco Corea's series: Big data strategy [Part 1](https://francesco-ai.medium.com/big-data-strategy-i-tips-for-analyzing-your-data-963292a43105) gives tips for analyzing your data with a goal in mind, [Part 2](https://francesco-ai.medium.com/big-data-strategy-part-ii-a-data-maturity-map-8e41db568625) is a data maturity map, and [Part 3](https://francesco-ai.medium.com/big-data-strategy-part-iii-is-your-company-data-driven-acf871c38001) asks whether your company is data-driven.

The last lenses are specific. [Data before ML Strategy](https://medium.com/machine-learning-in-practice/first-things-first-you-need-a-data-strategy-before-you-can-have-a-machine-learning-strategy-366f8439aedf) answers the CEOs who are asked for a machine learning or AI strategy: you need a data strategy first. Connect the Dots in Data Strategy looks at it [from a product POV](https://medium.com/data-science/connect-the-dots-in-data-strategy-56f65a1e63a5), Ivy Liu's story of a new business whose annual forecast disappointed because nobody had put all the business data in one place. The [Data Advantage Matrix](https://medium.com/data-science/data-advantage-matrix-a-new-way-to-think-about-data-strategy-4178cd2f520a) is Prukalpa's new way to think about data strategy. To go back to the start, [create an effective strategy](https://medium.com/method-perspectives/how-to-create-an-effective-data-strategy-ffedb1bdc71a) is the same Method guide as above, and [how to get started](https://medium.com/data-science/data-strategy-716059d5c066) is the same first read.

Three of those articles also exist at their original addresses. Adam Votava's Creating a strategy to put wind in your data sails, the create a data strategy piece, is at <https://towardsdatascience.com/data-strategy-make-data-count-fa42b42d409>. Ivy Liu's from a product POV piece, which starts "A few years ago, I worked in a new business in a public company", is at <https://towardsdatascience.com/connect-the-dots-in-data-strategy-56f65a1e63a5>. Prukalpa Sankar's Data Advantage Matrix, which says there's no one path to creating a data strategy, is at <https://towardsdatascience.com/data-advantage-matrix-a-new-way-to-think-about-data-strategy-4178cd2f520a>.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* (excellent) how to get started. This address no longer opens: <https://towardsdatascience.com/data-strategy-716059d5c066>
* (very good) research data strategy - for research orgs. This address no longer opens: <https://towardsdatascience.com/research-data-strategy-2fd447f83719>
* how to create a data strategy for your org. This address no longer opens: <https://towardsdatascience.com/how-to-create-a-data-strategy-for-your-organization-e0493110b2e7>
* how to create a perfect data strategy. This address no longer opens: <https://towardsdatascience.com/how-to-create-a-perfect-data-strategy-7e8fd9bbfad0>
* a step by step guide. This address no longer opens: <https://towardsdatascience.com/data-strategy-a-step-by-step-guide-91529a75c72b>
* Become Data-Driven or Perish: Why your company needs a Data Strategy and not just more Data People. This address no longer opens: <https://towardsdatascience.com/become-data-driven-or-perish-why-your-company-needs-a-data-strategy-and-not-just-more-data-people-aa5d435c2f9>
* how to create a successful data strategy. This address no longer opens: <https://towardsdatascience.com/how-to-create-a-successful-data-strategy-1293bacf463c>
* Definition taken from DAMA. This address no longer opens: <https://www.cleverrepublic.com/what-is/data-strategy/>
* what is data management. This address no longer opens: <https://www.cleverrepublic.com/what-is/data-management/>
* Data Strategy: Good Data vs. Bad Data. This address no longer opens: <https://towardsdatascience.com/data-strategy-good-data-vs-bad-data-d40f85d7ba4e>
* how to get started. This address no longer opens: <https://towardsdatascience.com/data-strategy-716059d5c066>


# Data Vision

A data team needs a vision that says what success looks like before the org chart is drawn. The page points at one worked example of that vision, written for a data team.

The vision comes before the team shape, and the team page picks up from here. The same notes are in [Data Teams](/data/engineering-intro/data-teams).

[Vision for a data team](https://medium.com/alan/vision-for-a-data-team-2eae845b8052) is Antoine Liz's write-up, opening with the line “Knowing where you want to go is the best way to arrive somewhere”, which is the whole argument for writing the vision first.


# Data Teams

Analytics only lands if the team shape can ship it. Once the vision is written, the page points at how to build an analytics team and then at how to set a data science team up for success.

The same notes are in [Building Data/DS teams](/business-problems/data-science#building-datads-teams), [Building Teams](/ai-product/management#building-teams), and [Data Vision](/data/engineering-intro/data-vision).

[How to build an effective analytics team](https://www.analytics8.com/blog/how-to-build-an-effective-data-analytics-team/) is Kevin Lobo's best practices for building and optimizing a high-impact data analytics team, with tips on being a successful data leader.

The analytics team is one shape; the data science team needs its own foundations. See also [Setting Up Data Science Teams For Success](https://pub.towardsai.net/setting-up-data-science-teams-for-success-b1bee9765517) (October 2022), built around a figure of the three pillars of data science.


# Data KPIs

Data spend needs a return measure, not only pipeline uptime. The page is about that one question, measuring the return on data and analytics.

The KPI for data sits inside the wider OKR and KPI practice of the organization. The same notes are in [Data Science OKR KPI](/ai-product/management#data-science-okr-kpi) and [OKRs & KPIs](/ai-product/management#okrs--kpis).

[How to measure ROI of data analytics](https://medium.com/be-data-driven/how-to-measure-the-roi-of-data-analytics-2d2ac172754a) is Bruno Aziza's piece on measuring the ROI of data analytics, in an issue that also asks whether thinking is the enemy of innovation and what to do when scaling a data team. A second source on measuring the ROI of data and analytics used to sit here; its address no longer opens and it is kept at the end of the page.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Measuring ROI of data and analytics. This address no longer opens: <https://santiagotacoronte.com/5-ways-to-measure-the-roi-of-data-analytics-the-simple-way-a61790eff7d1>


# Data Program Management

A data program needs roles and definitions that keep projects from drifting. The page starts with a definition of data program management, then moves to what a program manager does in general, and ends with the data program manager as a job.

Program management for data is a specialization of the general project and program practice. The same notes are in [Project & Program Management](/ai-product/project-and-program-management) and [Project Management](/ai-product/management#project-management).

The definition comes first. [What is a data program management? and a manager.](https://www.ddmcd.com/dpm-intro) by Dennis D McDonald. is his introduction to data program management (DPM), and <http://www.ddmcd.com/dpm> is his index of links to and excerpts from the site's articles on the topic.

A data program manager is first a program manager. <https://rebelsguidetopm.com/what-is-a-program-manager/#What_is_a_program> is a program manager explaining what a program is and what the role does, with the essential skills for anyone considering the career change; <https://rebelsguidetopm.com/what-is-a-program-manager/> is the same article from the top.

Then the data part. <https://www.velvetjobs.com/job-descriptions/data-program-manager> is a data program manager job description, which includes devOps support such as monitoring the health of data pipelines and escalating performance issues. <https://www.linkedin.com/pulse/how-become-data-program-manager-krishna-mohan-kaluri/> asks how to be a good data program manager and answers with a deep awareness of the data management life cycle, exceptional project management skills, and a solid grasp of benefits management. The list of [program manager responsibilities](https://www.projectmanager.com/training/program-manager-responsibilities) is Jennifer Bridges, PMP, with a full job description of the role.


# Data Engineering Questions

Interview loops and training prompts need a bank of questions, not a second textbook. The page starts with the general prompts, then points at larger question banks, and ends with the references the answers rest on, from CAP and ACID to join strategies. The same notes are in [Data Engineering](/data/engineering-intro/data-engineering).

## General

The general prompts come first, because they test experience before they test theory. The first eight come from LinkedIn's interviewing resources for data engineers, which cover interview strategies, the best questions for assessing a candidate, and how to improve the interview process; the rest are about BigQuery, Spark, and ETL.

1. [What scales of data have you worked with in the past?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
2. [How do you generally work with the departments that make use of your data?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
3. [Tell me about a time you had performance issues with an ETL. How did you identify this as a performance issue and how did you fix it?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
4. [Describe a time when you found a new use case for an existing database.](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
5. [Describe the most challenging project you’ve worked on. What was your role?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
6. [Think back to a project you’re proud of. What was it that gave you that sense of pride and accomplishment?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
7. [What do you consider to be one of the biggest mistakes you’ve ever made in your previous job?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
8. [Explain the differences between stream processing and data processing, with one caveat: pretend that I’m not familiar with data at all.](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
9. What are the considerations when choosing methods of ingesting data to BigQuery?
10. Have you worked with data science teams? What were your responsibilities?
11. What are the considerations of choosing Spark vs BigQuery?
12. What are the differences between ETL and ELT?

For the last question, [1](https://www.guru99.com/etl-vs-elt.html) is Guru99's ETL vs ELT comparison, and [2](https://www.xplenty.com/blog/etl-vs-elt/) is Mark Smallcombe's five critical differences, weighing cost, data type and volume, and other factors that shape the integration method. A third source used to sit here and is kept at the end of the page.

The same notes are in [Data Pipelines](/data/processing-intro/data-pipelines).

The three figures below draw the difference; the first is credited to guru99/david taylor and the third to mark smallcombe.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-17c1971fdb2cb14105380dbeab4a247ab6d8f610%2F0?alt=media" alt=""><figcaption><p>ETL and ELT</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-3a073f02d3c78f0411922895049f94c4715f1d3e%2F1?alt=media" alt=""><figcaption><p>ETL and ELT</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-24fd0fb1f88da56640cdf7840ff068381183a379%2F2?alt=media" alt=""><figcaption><p>ETL and ELT</p></figcaption></figure>

## More question banks

A dozen prompts is enough for one interview; training needs the larger banks.

The biggest is [More than 2000 questions for data engineers](https://github.com/OBenner/data-engineering-interview-questions), a GitHub repo of more than 2000 data engineer interview questions, shown in the figure below.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-e5c1aa9520ebc9f3138c012f96c9f3214afd341a%2F11?alt=media" alt=""><figcaption><p>More than 2000 questions for data engineers</p></figcaption></figure>

[More data engineering questions](https://realpython.com/data-engineer-interview-questions-python/) is Real Python's set of data engineer interview questions answered with Python.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-9eb39d05b1836c2b187f6946affc219934435868%2F12?alt=media" alt=""><figcaption><p>More data engineering questions</p></figcaption></figure>

[Even more qs](https://www.softwaretestinghelp.com/data-engineer-interview-questions/) is a list of the most frequently asked data engineer interview questions and answers, for preparing for an upcoming interview.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-dd8770e3e8e6b48e9619cc1120f77dd932dd5958%2F13?alt=media" alt=""><figcaption><p>Even more qs</p></figcaption></figure>

## References

Questions need answers, and the references are where the answers come from.

[DS leads](https://docs.google.com/document/d/1gdfJce0p7jx0ptHJt3NE3NIhvzZCXJ5O0V-dvJse0HI/edit) is a data engineer interviews document that opens with the CAP theorem and BigQuery questions, such as the difference in the implementation between partitions and clustering in BQ. For system design, the primer that teaches how to design large-scale systems has both [System design interview q’s with solutions](https://github.com/donnemartin/system-design-primer#system-design-interview-questions-with-solutions) and a [Cap theorem](https://github.com/donnemartin/system-design-primer#system-design-interview-questions-with-solutions) section. [2](https://github.com/henryr/cap-faq) is the CAP FAQ on GitHub. The (which is great) [3](https://github.com/kislerdm/data-engineering-interviews) is the data engineering interviews Q\&A by and for the data community, which great isn't complete.

The database guarantees are a three-part series. [ACID](https://bardoloi.com/blog/2017/02/26/db-deep-dive/) starts from how an amazing feat of engineering becomes commonplace and taken for granted, and lays out the ACID guarantees. [CAP](https://bardoloi.com/blog/2017/03/06/cap-theorem/) picks up from the ACID guarantees of traditional RDBMSs like MS SQL Server, Oracle, and MySQL, and shows where they run into scale. [PACLEC](https://bardoloi.com/blog/2017/03/06/pacelc-theorem/) takes the CAP theorem as the framework under most modern distributed databases and adds latency, following Daniel Abadi's critique of how CAP had been applied.

The last question is why the field exists at all. [Why do we need Data engineering?](https://podcastaddict.com/episode/116229803) (podcast) is the Data Engineering Podcast episode on proven patterns for building successful data teams, where many stakeholders with competing goals and many roles make data products hard to deliver.

The Spark join strategies article linked on the platforms page, Join strategies #2 from Towards Data Science, is also at its original address: <https://towardsdatascience.com/strategies-of-spark-join-c0e7b4572bcf>.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* 3. This address no longer opens: <https://blog.panoply.io/etl-vs-elt-the-difference-is-in-the-how>
* Can you explain the parquet file format? This address no longer opens: <https://parquet.apache.org/documentation/latest/>


# Next

Next is Problem Framing. The data is ready to be turned into a prediction problem. Once the table lives on one path from source to owner, the next step is to name the task and the label regime that table will be read under.


# Next

Next is Problem Framing. The four data chapters have produced a dataset a model can be asked a question about. The same table is a different problem under a different label regime, so the next part names the task and the learning setting before any model family is chosen.


# Introduction

The same table is a different problem under a different label regime, and there is no reinforcement-setting page here. The chapter first names the task, then walks the learning settings from weak and scarce labels to streams, few-shot data, and the choice of a model family. After this chapter the reader can name the task and the learning setting, and reinforcement learning is in [Reinforcement Learning](/decision-intelligence/reinforcement-learning).

## Tasks

The first step is to see where a problem sits among machine learning setups. The [Overview](/problem-framing/types-of-machine-learning) maps how those setups differ by how much supervision and feedback they use, so you can place a problem before you pick a method. When one example carries more than one label, [Multi Label Classification](/problem-framing/multi-label-classification) covers the main ways to assign several labels per example, with problem-transformation recipes, metrics, and tooling.

## Learning settings

Once the task is named, the label regime decides the setting. [Weakly Supervised](/problem-framing/weakly-supervised) is for labels that are incomplete, inaccurate, or inexact rather than fully supervised. [Label Algorithms](/problem-framing/label-algorithms) handles unbalanced labels, label propagation and spreading, and label noise, for when labels are scarce, noisy, or need to be spread across a graph of related examples. [Semi Supervised](/problem-framing/semi-supervised) is training with lots of unlabeled data and only a little labeled data. When labels are expensive, [Active Learning](/problem-framing/active-learning) chooses which examples a human labels next, with uncertainty and diversity sampling and query-by-committee. [Online Learning](/problem-framing/online-learning) is for models that update as labeled examples arrive in a stream, including non-stationary settings. [N-Shot Learning](/problem-framing/n-shot-learning) covers zero-, one-, and few-shot learning, adapting with only a handful of labeled examples per class. With the task and the setting known, [Model Families](/problem-framing/model-families) maps the problem shape to an algorithm family.


# Overview

Before any method is picked, a problem has to be placed: machine learning setups differ by how much supervision and feedback they use. The figure below is that map, and the rest of the chapter takes its regions one at a time.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-dd5f1d345146271943d474a893c25076b3f54593%2Fgimg-93c1ff1adb47.png?alt=media" alt=""><figcaption><p>Image via Abdul Rahid, via Dan Shewan, wrong credit? let me know</p></figcaption></figure>

The image comes via Abdul Rahid, via Dan Shewan, and is copied from the original hosted image; if that credit is wrong, let me know.


# Multi Label Classification

Some examples need more than one label, and a plain multiclass classifier cannot say that. The page starts with what multilabel classification is, then the methods and metrics, the problem-transformation recipes, an EfficientNet case, and the tooling.

The same notes are in [MULTI LABEL/OUTPUT](/deep-learning/deep-neural-nets#multi-labeloutput).

## What multilabel classification is

The starting point is the definition and its contrast with multiclass, taken from the mlr tutorial on multilabel classification. (what is?) [Multilabel classification is a classification problem where multiple target labels can be assigned to each observation instead of only one like in multiclass classification.](https://mlr.mlr-org.com/articles/tutorial/multilabel.html)

Two different approaches exist for multilabel classification. Problem transformation methods try to transform the multilabel classification into binary or multiclass classification problems. Algorithm adaptation methods adapt multiclass algorithms so they can be applied directly to the problem. I.e., the two approaches are to use a classifier that does multi label, or to use any classifier with a wrapper that compares each two labels.

## Methods and metrics

Once the two approaches are named, the next question is how each method works and how to score a prediction that is a set of labels. The same notes are in [Precision Recall ROC AUC](/evals/evaluation-metrics#precision--recall--roc--auc).

Jesse Read's two-part "Multi-label Classification" talk from Universidad Carlos III de Madrid is the great [PDF](https://users.ics.aalto.fi/jesse/talks/Multilabel-Part01.pdf) that explains about multi label classification and especially metrics, [part 2 here](https://users.ics.aalto.fi/jesse/talks/Multilabel-Part02.pdf) continues it. For the full catalogue, [An awesome Paper](https://www.researchgate.net/publication/273859036_Multi-Label_Classification_An_Overview) explains all of these methods in detail!

## Problem transformation methods

The paper's problem-transformation recipes are the concrete form of the first approach, and they are a genuine list, PT1 to PT6:

PT1: for each sample select one label, remove all others.

PT2: remove every sample which has multi labels.

PT3: for every combo of labels create a single-label, i.e. A\&B, A\&C etc.

PT4: (most common) create L datasets, for each label learn a binary representation, i.e., is it there or not.

PT5: duplicate each sample with only one of its labels

PT6: read the paper

There are other approaches for doing it within algorithms, they rely on the ideas PT3/4/5/6 implemented in the algorithms, or other tricks. They also introduce Label cardinality and label density.

## EfficientNet for multilabel

The recipes meet a real model in GumGum's threat-detection work, where brand safety for advertisers means scanning a publisher's inventory for threats that are often not the salient object in the image. [Efficient net](https://medium.com/gumgum-tech/multi-label-classification-for-threat-detection-part-1-60318b90ce11) is part 1 of that series, and [part 2](https://medium.com/gumgum-tech/multi-label-image-classifier-for-threat-detection-with-fp16-inference-part-2-40fe0f9a93b3), by Aditya Ramesh, runs exploratory data analysis on the multilabel dataset and FP16 inference. EfficientNet is based on a network derived from a neural architecture search and novel compound scaling method is applied to iteratively build more complex network which achieves state of the art accuracy on multiclass classification tasks. Compound scaling refers to increasing the network dimensions in all three scaling formats using a novel strategy.

## Tools

After the model, the work is in the libraries. Multi label confusion matrices with sklearn are the scikit-learn side, and the [Scikit multilearn package](http://scikit.ml/index.html), hosted at scikit.ml, is the scikit-multilearn side.

An awesome Paper that explains all of these methods in detail, also available here! <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.104.9401&rep=rep1&type=pdf>

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* (what is?) Multilabel classification is a classification problem where multiple target labels can be assigned to each observation instead of only one like in multiclass classification. This address no longer opens: <https://mlr-org.github.io/mlr-tutorial/devel/html/multilabel/index.html>


# Weakly Supervised

When labels are incomplete, inaccurate, or inexact rather than fully supervised, the model has to learn from weak signals. The page moves from a survey of that setting, to Snorkel and programmatic labeling, to papers on test-time training, noisy labels, and multiple instance learning (MIL), and ends with semi-weak supervision at scale.

The same notes are in [Label Propagation / Spreading](/problem-framing/label-algorithms#label-propagation--spreading) and [Semi Supervised](/problem-framing/semi-supervised).

The smallest case shows why weak supervision matters. Text classification with extremely small datasets relies heavily on feature engineering methods such as number of hashtags, number of punctuations and other insights that are really good for this type of text; the article behind that note is kept at the end of the page.

The map of the whole setting is a great [review paper](https://pdfs.semanticscholar.org/3adc/fd254b271bcc2fb7e2a62d750db17e6c2c08.pdf) from Nanjing University's National Key Laboratory for Novel Software Technology, great for weakly supervision. It discusses incomplete supervision, inaccurate, and inexact labels, and active learning.

With the survey's three kinds of weakness named, Stanford's answer is to write labels as programs. Stanford on weakly supervised learning had an older post, kept at the end of the page. [Stanford ai on snorkel](http://ai.stanford.edu/blog/weak-supervision/) is "Weak Supervision: A New Programming Paradigm for Machine Learning", which starts from deep learning models reaching state-of-the-art scores without hand-engineered features and asks where their training labels come from. [Intro to Snorkel](https://medium.com/@towardsai/data-centric-ai-with-snorkel-ai-the-enterprise-ai-platform-a8ed0803c24c) is "Data-Centric AI With Snorkel AI: The Enterprise AI Platform", which treats the data rather than the model specifics as the arbiter of whether AI deployment succeeds. Hazy research on weak and snorkel also had a post, kept at the end of the page.

Labels can also be weak at test time, noisy in training, or given only for groups. [Out of distribution generalization using test-time training](https://arxiv.org/abs/1909.13231) — the first case — "Test-time training turns a single unlabeled test instance into a self-supervised learning problem, on which we update the model parameters before making a prediction on this instance. "

[Learning Deep Networks from Noisy Labels with Dropout Regularization](https://arxiv.org/pdf/1705.03419.pdf) — the noisy-label case — "Large datasets often have unreliable labels—such as those obtained from Amazon’s Mechanical Turk or social media platforms—and classifiers trained on mislabeled datasets often exhibit poor performance. We present a simple, effective technique for accounting for label noise when training deep neural networks. We augment a standard deep network with a softmax layer that models the label noise statistics. Then, we train the deep network and noise model jointly via end-to-end stochastic gradient descent on the (perhaps mislabeled) dataset. The augmented model is overdetermined, so in order to encourage the learning of a non-trivial noise model, we apply dropout regularization to the weights of the noise model during training. Numerical experiments on noisy versions of the CIFAR-10 and MNIST datasets show that the proposed dropout technique outperforms state-of-the-art methods."

[Distill to label weakly supervised instance labeling using knowledge distillation](https://arxiv.org/pdf/1907.12926.pdf) — the inexact case, where only image-level labels exist — “Weakly supervised instance labeling using only image-level labels, in lieu of expensive fine-grained pixel annotations, is crucial in several applications including medical image analysis. In contrast to conventional instance segmentation scenarios in computer vision, the problems that we consider are characterized by a small number of training images and non-local patterns that lead to the diagnosis. In this paper, we explore the use of multiple instance learning (MIL) to design an instance label generator under this weakly supervised setting. Motivated by the observation that an MIL model can handle bags of varying sizes, we propose to repurpose an MIL model originally trained for bag-level classification to produce reliable predictions for single instances, i.e., bags of size 1. To this end, we introduce a novel regularization strategy based on virtual adversarial training for improving MIL training, and subsequently develop a knowledge distillation technique for repurposing the trained MIL model. Using empirical studies on colon cancer and breast cancer detection from histopathological images, we show that the proposed approach produces high-quality instance-level prediction and significantly outperforms state-of-the MIL methods.”

The last step joins weak labels with unlabeled data at scale. Researchers from Facebook AI Research (FAIR) have announced their work on a new concept in deep learning called Semi-weakly Supervised Learning; [Yet another article summarising FAIR](https://neurohive.io/en/state-of-the-art/semi-weakly-supervised-learning-increasing-classification-accuracy-with-billion-scale-unlabeled-images/) is Dane Mitrev's summary of how it increases classification accuracy with billion-scale unlabeled images.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Text classification with extremely small datasets. This address no longer opens: <https://towardsdatascience.com/text-classification-with-extremely-small-datasets-333d322caee2>
* Stanford on. This address no longer opens: <https://dawn.cs.stanford.edu/2017/07/16/weak-supervision/>
* Hazy research on weak and snorkel. This address no longer opens: <https://hazyresearch.github.io/snorkel/blog/ws\\_blog\\_post.html>


# Label Algorithms

Labels are often scarce, noisy, or unevenly spread, and each of those needs its own algorithm. The page takes them in that order: first unbalanced labels, then label propagation and spreading across a graph of related examples, then tools for label noise.

## Unbalanced labels

The first label problem is that one class dominates, and the fix can be resampling or a change to the loss. The same notes are in [Decision Trees](/predictive-ml/decision-trees), [IMBALANCED DATASETS](/data/datasets-intro/datasets#imbalanced-datasets), and [Introduction](/decision-intelligence/reinforcement-learning#introduction).

The resampling route is [imbalance learn](https://imbalanced-learn.org/stable/auto_examples/over-sampling/plot_comparison_over_sampling.html#sphx-glr-auto-examples-over-sampling-plot-comparison-over-sampling-py) — is an open-source, MIT-licensed library that provides tools when dealing with classification with imbalanced classes; the linked page is its comparison of over-sampling methods, shown in the figure below.

The loss route is [Classifying Job Titles With Noisy Labels Using REINFORCE](https://medium.com/@ziprecruiter.engineering/classifying-job-titles-with-noisy-labels-using-reinforce-ce1a4bde05e2) this article has a very nice trick in adding a reward component to the loss function in order to mitigate for unbalanced class label problem, instead of the usual balancing.

The same notes are in [Reinforcement Learning](/decision-intelligence/reinforcement-learning).

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Mgd48oS5_duTKOVE_Et%2F-MhRYgoaYSWj3cTRH8br%2F-MhRkTQF8tnqGQo22pbn%2Fimage.png?alt=media&amp;token=bd89c291-c02e-41b8-9dcc-fc293945397c" alt=""><figcaption><p>Imbalance Learn comparison.</p></figcaption></figure>

## Label Propagation / Spreading

Balancing assumes the labels exist; when only a few do, the labels have to travel to the unlabeled samples. This is semi-supervised label propagation and spreading on a graph Laplacian. The same notes are in [Semi Supervised](/problem-framing/semi-supervised) and [Weakly Supervised](/problem-framing/weakly-supervised).

Note: very much related to weakly and semi supervision, i.e., we have small amounts of labels and we want to generalize the labels to other samples, see also weak supervision methods.

Step 1 is to [build a laplacian](https://en.wikipedia.org/wiki/Laplacian_matrix) graph using KNN, distance metric is minkowski with p=2, i.e. euclidean distance. The NetworkX series walks the idea from there: the [Step by step tutorial](https://medium.com/@graphml/introduction-to-label-propagation-with-networkx-part-1-abcbe954a2e8) is "Introduction to Label Propagation with NetworkX — Part 1", which starts from the observation that similar objects, like two people with similar hobbies, are likely to form a connection, and [part 2](https://medium.com/@graphml/introduction-to-label-propagation-with-networkx-part-2-cd041fa44e1) implements Label Propagation in NetworkX.

[Spreading](https://scikit-learn.org/stable/modules/generated/sklearn.semi_supervised.LabelSpreading.html) (propagation upgrade), Essentially a community graph algorithm, however it resembles KNN in its nature, using semi supervised data set (i.e., labeled and unlabeled data) to spread or propagate labels to unlabeled data, with small incrementations in the algorithm, using KNN-like methodology, each unlabeled sample will be given a label based on its 1st order friends, if there is a tie, a random label is chosen. Nodes are connected by using a euclidean distance. The [Difference](https://www.researchgate.net/post/What_is_the_difference_between_Label_propagation_and_Label_spreading_in_semi-supervised_learning_context) between propagation and spreading is a laplacian matrix, vs normalized LM. The matrix itself is taught in [Laplacian matrix on youtube, videos 30-33](https://www.youtube.com/watch?v=siCPjpUtE0A\&list=PLLssT5z_DsK9JDLcT8T62VtzwyW9LNepV\&index=33), a Stanford lecture series that ends at lecture 33 on finding a spectral graph partition.

With the steps in place, the rest is worked examples and references. The [Really good example notebook](https://github.com/DavidBrear/sklearn-cookbook/blob/master/Chapter%204/4.1.1%20Label%20Propagation%20with%20Semi-Supervised%20Learning.ipynb) is chapter 4.1.1 of the code from the Scikit-Learn Cookbook, label propagation with semi-supervised learning. The [Spreading vs propagation](https://www.researchgate.net/post/What_is_the_difference_between_Label_propagation_and_Label_spreading_in_semi-supervised_learning_context) thread is the same ResearchGate question as above. The Wikipedia article on the label propagation algorithm is <https://en.wikipedia.org/wiki/Label_Propagation_Algorithm>.

For video, Youtube [1](https://www.youtube.com/watch?v=UWf8hxeehOg) is Graphileon's community detection by label propagation, and [2](https://www.youtube.com/watch?v=hmashUPJwSQ) and [3](https://www.youtube.com/watch?v=F4f247IyOTs) are Leonid Zhukov's Network Analysis lecture 17, parts 1 and 2, on label propagation on graphs. Part 1 of the NetworkX series is also on [Medium](https://medium.com/@graphml/introduction-to-label-propagation-with-networkx-part-1-abcbe954a2e8).

In scikit-learn, the LabelPropagation page of the Sklearn user guide [1](https://scikit-learn.org/stable/modules/generated/sklearn.semi_supervised.LabelPropagation.html) and [2](https://scikit-learn.org/stable/modules/generated/sklearn.semi_supervised.LabelPropagation.html) is the API, and [3](https://scikit-learn.org/stable/auto_examples/semi_supervised/plot_label_propagation_digits.html) is the digits example, which demonstrates the power of semisupervised learning by training a Label Spreading model to classify handwritten digits with sets of very few labels.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-a80d1556ae89fe2e9a756465dffdcc4ef39b0a36%2Fgimg-ef14eddcbf6d.png?alt=media" alt=""><figcaption><p>Label propagation.</p></figcaption></figure>

The figure shows the basic propagation; the implementations go further. [Git](https://github.com/benedekrozemberczki/LabelPropagation) is a NetworkX implementation of Label Propagation from a "Near Linear Time Algorithm to Detect Community Structures in Large-Scale Networks" (Physical Review E 2008). [incremental LP](https://github.com/johny-c/incremental-label-propagation) is Incremental Label Propagation (ILP), incremental semi-supervised learning from streams for object classification. [Git2](https://github.com/yamaguchiyuto/label_propagation/) holds implementations of label propagation like algorithms:

1. Harmonic Function (HMN) \[Zhu+, ICML03]
2. Local and Global Consistency (LGC) \[Zhou+, NIPS04]
3. Partially Absorbing Random Walk (PARW) \[Wu+, NIPS12]
4. OMNI-Prop (OMNIProp) \[Yamaguchi+, AAAI15]
5. Confidence-Aware Modulated Label Propagation (CAMLP) \[Yamaguchi+, SDM16]

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-22d2d647d95009c0412557dbe16c37744fa09c00%2Fgimg-cfbad8f6ef70.png?alt=media" alt=""><figcaption><p>Label propagation variants.</p></figcaption></figure>

For slides, Presentation [1](http://www.leonidzhukov.net/hse/2015/networks/lectures/lecture17.pdf) is Zhukov's "Label propagation on graphs" lecture notes from the School of Data Analysis and Artificial Intelligence, and [2](https://www.slideshare.net/dav009/label-propagation-semisupervised-learning-with-applications-to-nlp) is "Label propagation - Semisupervised Learning with Applications to NLP".

Neo4j 1, 2, 3, were the Neo4j label-propagation pointers; the first is kept at the end of the page.

## Label Noise

Propagation spreads labels as if they were right; the last problem is labels that are wrong. This is cleanlab, PU-learning, and pumml for positive-unlabeled data. The same notes are in [Annotation & Disagreement](/data/processing-intro/annotation-and-disagreement).

The main tool is [clean lab](https://github.com/cleanlab/cleanlab), the open-source library for data quality and machine learning with messy, real-world data and labels. Its own description — "`cleanlab` is the data-centric ML ops package for machine learning with noisy labels. `cleanlab` `clean`s `lab`els and supports finding, quantifying, and learning with label errors in datasets. See datasets cleaned with `cleanlab` at [labelerrors.com](https://labelerrors.com/). Check out the: [cleanlab code documentation](https://cleanlab.readthedocs.io/). `cleanlab` is powered by confident learning, published in this [paper](https://jair.org/index.php/jair/article/view/12125) | [blog](https://l7.curtisnorthcutt.com/confident-learning)." The labelerrors.com site shows label errors in benchmark ML datasets, the paper is Curtis Northcutt's confident learning article, and the blog overviews that paper by Curtis G. Northcutt, Lu Jiang, and Isaac L. Chuang.

The research behind it has two references. Reference 1: [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://openreview.net/forum?id=XccDXrDNLek), by [Curtis G Northcutt](https://openreview.net/profile?id=~Curtis_G_Northcutt1), [Anish Athalye](https://openreview.net/profile?id=~Anish_Athalye1), and [Jonas Mueller](https://openreview.net/profile?id=~Jonas_Mueller1). Reference 2: [Confident Learning: Estimating Uncertainty in Dataset Labels](https://www.jair.org/index.php/jair/article/view/12125) in the Journal of Artificial Intelligence Research, which moves confidence from model predictions to label quality by pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples. It can be used for positive unlabeled learning.

That last point opens the positive-unlabeled case. [PULearn](https://github.com/pulearn/pulearn) — "Positive-unlabeled learning (aka PU-learning) is a machine learning scenario for binary classification where the training set consists of a set of positively-labeled examples and an additional unlabeled set that contains positive and negative examples in unknown proportions (so no training example is explicitly labeled as negative). Positive-unlabeled learning methods aim to incorporate the unique structure of this scenario into the learning process, in a way that improves generalization of the learned notion of the positive class, when compared to simply treating all unlabeled examples as negative examples, or alternatively discarding them and training a one-class classifier over only the positive samples."

[PUMML](https://github.com/ncfrey/pumml), Medium — "Positive and Unlabeled Materials Machine Learning (pumml) is a code that uses semi-supervised machine learning to classify materials from only positive and unlabeled examples." The Medium post is <https://towardsdatascience.com/positive-and-unlabeled-materials-machine-learning-8b216edea899>, which describes that implementation for predicting the synthesizability of theoretical materials.

Back on propagation, reference 4 (plotly) was the Plotly version of the label-propagation structure example, <https://plot.ly/scikit-learn/plot-label-propagation-structure/>; the page now says it does not exist.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Sklearn. This address no longer opens: <https://scikit-learn.org/stable/modules/label\\_propagation.html>
* 1. This address no longer opens: <https://dzone.com/articles/graph-algorithms-in-neo4j-label-propagation>


# Semi Supervised

With lots of unlabeled data and only a little labeled data, the question is how to make the unlabeled part teach the model. The page starts with surveys, then self-training and tri-training, then consistency-based methods such as UDA, MixMatch, ReMixMatch, and FixMatch, and ends with pseudo-labeling and student-teacher approaches.

The same notes are in [COP-CLUSTERING](/predictive-ml/clustering-algorithms#cop-clustering), [Label Propagation / Spreading](/problem-framing/label-algorithms#label-propagation--spreading), and [Weakly Supervised](/problem-framing/weakly-supervised).

The map comes first. The [Paper review](https://pdfs.semanticscholar.org/3adc/fd254b271bcc2fb7e2a62d750db17e6c2c08.pdf) is Zhi-Hua Zhou's survey from Nanjing University's National Key Laboratory for Novel Software Technology, the same review that frames weak supervision. [Ruder an overview of proxy labeled for semi supervised (AMAZING)](https://ruder.io/semi-supervised/) starts from the point that while unsupervised learning is still elusive, researchers have made a lot of progress in semi-supervised learning, and focuses on methods that assign proxy labels to unlabelled data and use them as targets for learning.

Self training is the simplest proxy-label method: the model labels unlabeled data and learns from its own labels. [Self training and tri training](https://github.com/zidik/Self-labeled-techniques-for-semi-supervised-learning) is the zidik repository of self-labeled techniques. [Confidence regularized self training](https://github.com/yzou2/CRST) is the code for Confidence Regularized Self-Training, an ICCV19 oral. [Domain adaptation for semantic segmentation using class balanced self-training](https://github.com/yzou2/CBST) is the code for the ECCV18 paper on class-balanced self-training. The same zidik repository also serves as the reference for [Self labeled techniques for semi supervised learning](https://github.com/zidik/Self-labeled-techniques-for-semi-supervised-learning).

Tri training replaces one self-labeling model with three that label for each other. [Trinet for semi supervised Deep learning](https://www.ijcai.org/Proceedings/2018/0278.pdf) is the Tri-net paper from Nanjing University's National Key Laboratory for Novel Software Technology. [Tri training exploiting unlabeled data using 3 classes](https://www.researchgate.net/publication/3297469_Tri-training_Exploiting_unlabeled_data_using_three_classifiers) is the tri-training paper on exploiting unlabeled data using three classifiers. [Improving tri training with unlabeled data](https://link.springer.com/chapter/10.1007/978-3-642-25349-2_19) is a chapter in Software Engineering and Knowledge Engineering (AISC 115), and [Tri training using NN ensemble](https://link.springer.com/chapter/10.1007/978-3-642-31919-8_6) is the Springer chapter on doing it with a neural-network ensemble. [Asymmetric try training for unsupervised domain adaptation](https://github.com/corenel/pytorch-atda) is a PyTorch implementation of Asymmetric Tri-training for Unsupervised Domain Adaptation, [another implementation](https://github.com/vtddggg/ATDA) is an unofficial one, and [another](https://github.com/ksaito-ut/atda) is the TensorFlow version, all for the arXiv 1702.08400 [paper](https://arxiv.org/abs/1702.08400). [Tri training git](https://github.com/LiangjunFeng/Tri-training) is the python implementation of tri-triaing.

The newer methods add consistency and augmentation on top of proxy labels. If you have access to lots of unlabeled data, but a relatively small amount of labeled data, Semi-Supervised Learning (SSL) might be really useful; the [Fast ai forums](https://forums.fast.ai/t/semi-supervised-learning-ssl-uda-mixmatch-s4l/56826) thread on UDA, MixMatch, and S4L is the entry point. [UDA GIT](https://github.com/google-research/uda) is Google Research's Unsupervised Data Augmentation (UDA) code, and its [paper](https://arxiv.org/abs/1904.12848) is "Unsupervised Data Augmentation for Consistency Training". [medium\*](https://medium.com/syncedreview/google-brain-cmu-advance-unsupervised-data-augmentation-for-ssl-c0a6157505ce) is Synced's write-up of how Google Brain and CMU advance unsupervised data augmentation for semi-supervised learning, starting from data augmentation as a way to expand labeled training data. Unsupervised Data Augmentation is also covered in medium 2 ( [has data augmentation articles)](https://medium.com/towards-artificial-intelligence/unsupervised-data-augmentation-6760456db143), which starts from how costly it is to annotate a large amount of training data. [s4l](https://arxiv.org/abs/1905.03670) is "S4L: Self-Supervised Semi-Supervised Learning".

[Google’s UDM and MixMatch dissected](https://mlexplained.com/2019/06/02/papers-dissected-mixmatch-a-holistic-approach-to-semi-supervised-learning-and-unsupervised-data-augmentation-explained/) explains the text side of UDA: - For text classification, the authors used a combination of back translation and a new method called TF-IDF based word replacing.

Back translation consists of translating a sentence into some other intermediate language (e.g. French) and then translating it back to the original language (English in this case). The authors trained an English-to-French and French-to-English system on the WMT 14 corpus.

TF-IDF word replacement replaces words in a sentence at random based on the TF-IDF scores of each word (words with a lower TF-IDF have a higher probability of being replaced).

MixMatch is the next step after UDA. [MixMatch](https://arxiv.org/abs/1905.02249) is "MixMatch: A Holistic Approach to Semi-Supervised Learning"; it has a medium post kept at the end of the page, and [2](https://medium.com/@sanjeev.vadiraj/eureka-mixmatch-a-holistic-approach-to-semi-supervised-learning-125b14e82d2f), [3](https://medium.com/@sshleifer/mixmatch-paper-summary-1995f3d11cf), [4](https://medium.com/@literallywords/tl-dr-papers-mixmatch-9dc4cd217121) are summaries of it: "Eureka", which frames SSL as using extra information about the input distribution; a readable paper summary for training on little labeled and much unlabeled data; and a TL;DR that calls it a form of data augmentation and pseudo-labeling combining already-known techniques. The algorithm is one that works by guessing low-entropy labels for data-augmented unlabeled examples and mixing labeled and unlabeled data using MixUp. We show that MixMatch obtains state-of-the-art results by a large margin across many datasets and labeled data amounts

ReMixMatch - [paper](https://arxiv.org/pdf/1911.09785.pdf) is really good. “We improve the recently-proposed “MixMatch” semi-supervised learning algorithm by introducing two new techniques: distribution alignment and augmentation anchoring”

[FixMatch](https://amitness.com/2020/03/fixmatch-semi-supervised/) - FixMatch is a recent semi-supervised approach by Sohn et al. from Google Brain that improved the state of the art in semi-supervised learning(SSL). It is a simpler combination of previous methods such as UDA and ReMixMatch. The figure below shows it.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-0fbc4e6c9a488c070ca489f4621939965a56389b%2Fgimg-86a0a567680e.png?alt=media" alt=""><figcaption><p>FixMatch semi-supervised learning.</p></figcaption></figure>

The FixMatch figure is via Amit Chaudhary. Pseudo-labeling is revisited in [Curriculum Labeling: Self-paced Pseudo-Labeling for Semi-Supervised Learning](https://arxiv.org/pdf/2001.06001.pdf), which applies pseudo-labels to the unlabeled set using a model trained on the small labeled set.

At scale, the proxy labels come from a teacher model. [FAIR](https://ai.facebook.com/blog/billion-scale-semi-supervised-learning/) is Facebook AI's billion-scale semi-supervised learning for image and video classification, part of doing more with less labeled training data overall. [2](https://ai.facebook.com/blog/mapping-the-world-to-help-aid-workers-with-weakly-semi-supervised-learning/) is the weakly, semi-supervised mapping work, where Facebook AI researchers created the world’s most detailed population density maps of Africa to help humanitarian organizations. The original [Summarization of FAIR’s student teacher weak/ semi supervision](https://analyticsindiamag.com/how-to-do-machine-learning-when-data-is-unlabelled/) was on AIM — India. For text, [Leveraging Just a Few Keywords for Fine-Grained Aspect Detection Through Weakly Supervised Co-Training](https://www.aclweb.org/anthology/D19-1468.pdf) is the EMNLP-IJCNLP 2019 paper from Hong Kong.

[Fidelity-Weighted](https://openreview.net/forum?id=B1X0mzZCW) Learning - “fidelity-weighted learning” (FWL), a semi-supervised student- teacher approach for training deep neural networks using weakly-labeled data. FWL modulates the parameter updates to a student network (trained on the task we care about) on a per-sample basis according to the posterior confidence of its label-quality estimated by a teacher (who has access to the high-quality labels). Both student and teacher are learned from the data."

The student-teacher code follows. [Unproven student teacher git](https://github.com/EricHe98/Teacher-Student-Training) stores the files from a summer internship on "teacher-student learning", an experimental method for training deep neural networks using a trained teacher model. [A really nice student teacher git with examples](https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation) is the CVPR2020 oral code for Revisiting Knowledge Distillation via Label Smoothing Regularization, shown in the figure below.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-d409b5239e0dd1f468ade11800545b969b287e09%2Fgimg-26d0628511e9.png?alt=media" alt=""><figcaption><p>Image by yuanli2333. wrong credit? let me know</p></figcaption></figure>

The student-teacher idea then closes the loop back to tri-training: [Teacher student for tri training for unlabeled data exploitation](https://arxiv.org/abs/1909.11233) is "Teacher-Student Learning Paradigm for Tri-training: An Efficient Method for Unlabeled Data Exploitation". The figure below is by the late Dr. Hui Li, @ SAS.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-840ea1b76133d1c815f322d05f6d54958c63df40%2Fgimg-197804bc1a34.png?alt=media" alt=""><figcaption><p>Image by the late Dr. Hui Li, @ SAS. wrong credit? let me know</p></figcaption></figure>

The last paper states the setting of the whole page: in many practical data mining applications such as web page classification, unlabeled training examples are readily available but labeled ones are fairly expensive to obtain, which is why semi-supervised algorithms such as co-training exist. The paper is <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.487.2431&rep=rep1&type=pdf>.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* medium. This address no longer opens: <https://towardsdatascience.com/a-fastai-pytorch-implementation-of-mixmatch-314bb30d0f99>


# Active Learning

When labels are expensive, the model should choose which examples a human labels next. The page starts with tutorials and surveys, then the warnings and pitfalls around query strategies such as query-by-committee, then tools and videos, and ends with Robert Munro's human-in-the-loop book and its uncertainty and diversity sampling cheat sheets.

The same notes are in [Active Learning Algorithms](/decision-intelligence/active-learning-algorithms) and [Annotation & Disagreement](/data/processing-intro/annotation-and-disagreement).

If you need to start somewhere start [here](https://www.datacamp.com/community/tutorials/active-learning) - types of AL, the methodology, examples, sample selection functions. After that tutorial, the reference is Burr Settles' "Active Learning Literature Survey", Computer Sciences Technical Report 1648 at the University of Wisconsin–Madison, updated on January 26, 2010. It is a thorough thorough about [review paper](http://burrsettles.com/pub/settles.activelearning.pdf), and in practice [The book on AL](http://burrsettles.com/pub/settles.activelearning.pdf).

The survey promises cheaper labels, and the first warning is that the promise is conditional. [Choose your model first, then do AL, from lighttag](https://www.lighttag.io/blog/active-learning-optimization-is-not-imporvement/) is LightTag's "Active Learning: Optimization Is Not Improvement": optimizing model uncertainty does not necessarily improve the dataset or the business outcome. The alternative is Query by committee - Importantly, the active learning method we presented above is the most naive form of what is called "uncertainty sampling" where we chose to sample based on how uncertain our model was. An alternative approach, called Query by Committee, maintains a collection of models (the committee) and selecting the most "controversial" data point to label next, that is one where the models disagreed on. Using such a committee may allow us to overcome the restricted hypothesis a single model can express, though at the onset of a task we still have no way of knowing what hypothesis we should be using.

The same notes are in [Ensembles](/predictive-ml/ensembles) and [Entropy / Information Gain](/data/analytics-intro/information-theory#entropy--information-gain).

The same model dependence shows up in the papers. This [Paper](https://arxiv.org/pdf/1807.04801.pdf): warning against transferring actively sampled datasets to other models. [How to increase accuracy with AL ](http://www.ijcte.org/papers/910-AC0013.pdf)starts from the point that more labeled samples give more success, but labels are costly and slow to obtain, which is what active learning algorithms emerged to overcome. [AL with model selection](https://ojs.aaai.org/index.php/AAAI/article/download/9014/8873) - paper, from Carnegie Mellon's Machine Learning Department. Using weak and strong oracle in AL is the Chalmers University of Technology [paper](http://publications.lib.chalmers.se/records/fulltext/248447/248447.pdf) from the Department of Signals and Systems.

Those papers each fix one choice; the pitfalls paper lists all of them. [The pitfalls of AL](http://www.kdd.org/exploration_files/v12-02-9-UR-Attenberg.pdf) - how to choose (cost-effectively) the active learning technique when one starts without the labeled data needed for methods like cross-validation; 2. how to choose (cost-effectively) the base learning technique when one starts without the labeled data needed for methods like cross-validation, given that we know that learning curves cross, and given possible interactions between active learning technique and base learner; 3. how to deal with highly skewed class distributions, where active learning strategies find few (or no) instances of rare classes; 4. how to deal with concepts including very small subconcepts (“disjuncts”)—which are hard enough to find with random sampling (because of their rarity), but active learning strategies can actually avoid finding them if they are misclassified strongly to begin with; 5. how best to address the cold-start problem, and especially 6. whether and what alternatives exist for using human resources to improve learning, that may be more cost efficient than using humans simply for labeling selected cases, such as guided learning \[3], active dual supervision \[2], guided feature labeling \[1], etc.

Knowing when to stop is its own question: the [Confidence based stopping criteria paper](http://www.cs.cmu.edu/~./hovy/papers/10ACMjournal-activelearning-stopping.pdf) is "Confidence-Based Stopping Criteria for Active Learning for Data Annotation" by Jingbo Zhu and Huizhen Wang (Northeastern University), Eduard Hovy (University of Southern California), and Matthew Ma (Scientific Works). The setting behind all of this is that a lot of unlabeled data is plentiful and cheap, eg. documents off the web, speech samples, images and video, as a great [tutorial ](http://hunch.net/~active_learning/active_learning_icml09.pdf)puts it.

From papers to working systems: [AWS Sagemaker Active Learning](https://youtu.be/8J7y513oSsE?t=435), using annotation consolidation that finds outliers and weights accordingly, then takes that data, trains a model with the annotation + training data, if labeled with high probability, will use those labels, otherwise will re-annotate. For a first look on video, there is [An ok video](https://www.youtube.com/watch?v=Et7h1A1j4ns\&feature=youtu.be). For code, [Active learning framework in python](https://github.com/bwallace/curious_snake) is bwallace's curious\_snake. [Active Learning Using Pre-clustering](https://www.researchgate.net/profile/Arnold_Smeulders/publication/221345455_Active_learning_using_pre-clustering/links/54c3cc440cf2911c7a4cc74a/Active-learning-using-pre-clustering.pdf) clusters before it queries. For language, [A literature survey of active machine learning in the context of natural language processing](https://www.ccs.neu.edu/home/vip/teach/MLcourse/4_boosting/materials/SICS-T--2009-06--SE.pdf) is the NLP survey. Practical Online Active Learning for Classification is the online version, linked at the end of the body. [Video 2](https://www.youtube.com/watch?v=8Jwp4_WbRio\&index=7\&list=PLegWUnz91Wfsn6skGOofRoeFoOyfdqSyN) is a second video. [Active learning in R - code](https://github.com/gsimchoni/ActiveLearningExercise) is an exercise in active learning implementation in R on the MNIST dataset. [Deep bayesian active learning with image data](https://arxiv.org/pdf/1703.02910.pdf) takes on the difficulty that AL methods rely on learning and updating models from small amounts of data, which deep learning tools do not do well.

Active learning puts a human back in the loop. [Integrating Human-in-the-Loop (HITL) in machine learning is a necessity, not a choice. Here’s why?](https://medium.com/@supriya2211/integrating-human-in-the-loop-hitl-in-machine-learning-application-is-a-necessity-not-a-choice-f25e131ca84e) is the case for it, by Andy Kelly, and the figure below is its basic framework for HITL, credited to Supriya Ghosh.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2FS1lP1l4udlJDsPoBZuTq%2Fimage.png?alt=media&amp;token=db317186-e311-4afc-b713-5f2af3f11612" alt=""><figcaption><p>Basic Framework for HITL Supriya Ghosh wrong credit? let me know</p></figcaption></figure>

#### Human In The loop ML book by [Robert munro](https://www.manning.com/books/human-in-the-loop-machine-learning#ref)

The HITL framework has a full book behind it: Robert (Munro) Monarch's Human-in-the-Loop Machine Learning, with related cheat sheets. [GIT](https://github.com/rmunro/pytorch_active_learning) is the PyTorch library for active learning that accompanies the book. [Active transfer learning](https://medium.com/pytorch/active-transfer-learning-with-pytorch-71ed889f08c1) is "Active Transfer Learning with PyTorch": models adapted to predict their own errors, combining active learning and transfer learning.

The first cheat sheet is about confidence. When a Supervised Machine Learning model makes a prediction, it often gives a confidence in that prediction; if the model is uncertain, human feedback can help, and that is [Uncertainty sampling](https://medium.com/data-science/uncertainty-sampling-cheatsheet-ec57bc067c0b). Its four measures are:

* Least Confidence: difference between the most confident prediction and 100% confidence
* Margin of Confidence: difference between the top two most confident predictions
* Ratio of Confidence: ratio between the top two most confident predictions
* Entropy: difference between all predictions, as defined by information theory

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-af947dbeaa5c0bb415bf7b4ae0c9c43f1a5796c8%2Fgimg-2b1ae57527ed.png?alt=media" alt=""><figcaption><p>Uncertainty sampling.</p></figcaption></figure>

Uncertainty alone finds what confuses the model, not what is missing from the data. [Diversity sampling](https://medium.com/data-science/diversity-sampling-cheatsheet-32619693c304) - you want to make sure that it covers as diverse a set of data and real-world demographics as possible. Its four strategies are:

* Model-based Outliers: sampling for low activation in your logits and hidden layers to find items that are confusing to your model because of lack of information
* Cluster-based Sampling: using Unsupervised Machine Learning to sample data from all the meaningful trends in your data’s feature-space
* Representative Sampling: sampling items that are the most representative of the target domain for your model, relative to your current training data
* Real-world diversity: using sampling strategies that increase fairness when trying to support real-world diversity

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-a523558d94bfb3892939723671ad6ea35ada1405%2Fgimg-0d79b6e997b7.png?alt=media" alt=""><figcaption><p>Diversity sampling.</p></figcaption></figure>

The two are strongest together. [Combine uncertainty sampling and diversity sampling](https://medium.com/data-science/advanced-active-learning-cheatsheet-d6710cba7667) is the "Advanced Active Learning Cheatsheet", on selecting the optimal unlabeled data for a human to review when systems are trained on thousands or millions of human-labeled examples. Its ten combinations:

1. Least Confidence Sampling with Clustering-based Sampling: sample items that are confusing to your model and then cluster those items to ensure a diverse sample (see diagram below).
2. Uncertainty Sampling with Model-based Outliers: sample items that are confusing to your model and within those find items with low activation in the model.
3. Uncertainty Sampling with Model-based Outliers and Clustering: combine methods 1 and 2.
4. Representative Cluster-based Sampling: cluster your data to capture multinodal distributions and sample items that are most like your target domain (see diagram below).
5. Sampling from the Highest Entropy Cluster: cluster your unlabeled data and find the cluster with the highest average confusion for your model.
6. Uncertainty Sampling and Representative Sampling: sample items that are both confusing to your current model and the most like your target domain.
7. Model-based Outliers and Representative Sampling: sample items that have low activation in your model but are relatively common in your target domain.
8. Clustering with itself for hierarchical clusters: recursively cluster to maximize the diversity.
9. Sampling from the Highest Entropy Cluster with Margin of Confidence Sampling: find the cluster with the most confusion and then sample for the maximum pairwise label confusion within that cluster.
10. Combining Ensemble Methods and Dropouts with individual strategies: aggregate results that come from multiple models or multiple predictions from one model via Monte-Carlo Dropouts aka Bayesian Deep Learning.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-5cf8afb7981e90fb4641811f968450bbc51f3ec7%2Fgimg-7d3db8ffec5a.png?alt=media" alt=""><figcaption><p>Combine uncertainty sampling and diversity sampling.</p></figcaption></figure>

Active transfer learning, from the PyTorch article above, is the next diagram.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-efe92d40d22a4ce969d375654d74d89fa9a8be39%2Fgimg-302565cf235b.png?alt=media" alt=""><figcaption><p>Active transfer learning.</p></figcaption></figure>

Machine in the loop flips the roles: the model suggests and the human confirms. LightTag's "When Should You Use Machine in the Loop?" explains that machine-in-the-loop labeling can reduce annotation cost while introducing bias, and how to reason about the tradeoff: [Similar to AL, just a machine / model / algo adds suggestions. This is obviously a tradeoff of bias and clean dataset](https://www.lighttag.io/blog/when-to-use-machine-in-the-loop/)

Practical Online Active Learning for Classification, from the Department of Computer Science and Engineering at the University of California, San Diego, is <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.87.5536&rep=rep1&type=pdf>

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* AL with model selection - paper. This address no longer opens: <http://www.alnurali.com/papers/paper\\_aaai\\_2014.pdf>
* A literature survey of active machine learning in the context of natural language processing. This address no longer opens: <http://eprints.sics.se/3600/>
* Mnist competition (unpublished) using AL. This address no longer opens: <http://dag.cvc.uab.es/mnist/statistics/>
* Medium on AL. This address no longer opens: <https://news.voyage.auto/active-learning-and-why-not-all-data-is-created-equal-8a43a758c6f9>
* Uncertainty sampling. This address no longer opens: <https://towardsdatascience.com/uncertainty-sampling-cheatsheet-ec57bc067c0b>
* Diversity sampling - you want to make sure that it covers as diverse a set of data and real-world demographics as possible. This address no longer opens: <https://towardsdatascience.com/https-towardsdatascience-com-diversity-sampling-cheatsheet-32619693c304>
* Combine uncertainty sampling and diversity sampling. This address no longer opens: <https://towardsdatascience.com/advanced-active-learning-cheatsheet-d6710cba7667>


# Online Learning

Some models cannot wait for a finished dataset: they must update as labeled examples arrive in a stream, including non-stationary settings. The page starts with classical online learning, its tutorials, theory, and tools, and then moves to online deep learning.

The same notes are in [Active Learning Algorithms](/decision-intelligence/active-learning-algorithms), [Follow the regularized leader](/decision-intelligence/follow-the-regularized-leader), [Incremental Learning](/decision-intelligence/incremental-learning), and [Training Strategies](/evals/training-strategies).

### Online (Classical) Learning

Classical online learning updates the model as labeled examples arrive in a stream, instead of building one model on the whole dataset at once. If you want to start with OL, [start here](https://dziganto.github.io/data%20science/online%20learning/python/scikit-learn/An-Introduction-To-Online-Machine-Learning/): "An Introduction To Online Machine Learning" begins from batch, or offline, learning, the standard approach you already know, and contrasts it with the online version. Analytics Vidhya's "Introduction to Online Machine Learning: Simplified" is the second introduction, [here](https://www.analyticsvidhya.com/blog/2015/01/introduction-online-machine-learning-simplified-2/).

The theory is Shay Shalev's. [A thesis about online learning](http://ttic.uchicago.edu/~shai/papers/ShalevThesis07.pdf) is his thesis, opening with “How much better to get wisdom than gold, to choose understanding rather than silver”. [Some answers about what is OL,](https://www.quora.com/What-is-the-best-way-to-learn-online-machine-learning) are the Quora answers to "What is the best way to learn online machine learning?", which frame it as models that learn incrementally from data streams and point at production concerns such as latency, concept drift, and evaluation under streaming; the first one actually talks about S.Shalev’s [other paper.](http://www.cs.huji.ac.il/~shais/papers/OLsurvey.pdf), his online learning survey. Online learning — Andrew Ng — coursera is the lecture version, kept at the end of the page.

From theory to production: [Chip Huyen on online prediction & learning](https://huyenchip.com/2020/12/27/real-time-machine-learning.html) argues that machine learning is going real-time, with a 2022 follow-up on its challenges and solutions. For code, [River](https://github.com/online-ml/river/) is a Python library for [online machine learning](https://www.wikiwand.com/en/Online_machine_learning), and the figure below sketches the setting.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2FmmrGK37xlBVxTNwNQKst%2Fimage.png?alt=media&amp;token=e3519d9b-e88d-4048-9ab3-205a1f0529f1" alt=""><figcaption><p>Online machine learning.</p></figcaption></figure>

### Online Deep Learning (ODL)

After the classical setting, online deep learning adapts neural models from streaming or non-stationary data. The notes here name Hedge back propagation (HDP), Autonomous DL, Qactor — online AL for noisy labeled stream data. The article that covered them is kept at the end of the page.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Hedge back propagation (HDP), Autonomous DL, Qactor. This address no longer opens: <https://towardsdatascience.com/online-deep-learning-odl-and-hedge-back-propagation-277f338a14b2>
* coursera. This address no longer opens: <https://www.coursera.org/learn/machine-learning/lecture/ABO2q/online-learning>


# N-Shot Learning

Sometimes each class has only a handful of labeled examples, or none at all. The page defines zero-, one-, and few-shot learning, then shows how zero-shot learning replaces labels with class vectors, and ends with how large language models treat prompts as zero-, one-, and few-shot shots.

The same notes are in [SIAMESE NETWORKS](/deep-learning/deep-neural-nets#siamese-networks) and [SIAMESE NETWORKS (one shot)](/deep-learning/siamese-nets#siamese-networks-one-shot).

### N-SHOT LEARNING

N-shot learning means training or adapting with only a handful of labeled examples per class. The FloydHub Team's "N-Shot Learning: Learning More with Less Data" asks whether it is possible to use machine learning with small data, and answers yes, covering [Zero shot, one shot, few shot](https://floydhub.github.io/n-shot-learning/).

### ZERO SHOT LEARNING

The extreme case is zero examples: zero-shot learning classifies new classes using side information instead of labeled examples for those classes. [Instead of using class labels](https://www.youtube.com/watch?v=jBnCcr-3bXc), we use some kind of vector representation for the classes, taken from a co-occurrence-after-svd or word2vec. — quite clever. This enables us to figure out if a new unseen class is near one of the known supervised classes. KNN can be used or some other distance-based classifier. Can we use word2vec for similarity measurements of new classes?

The same notes are in [WORD2VEC](/natural-language-processing-nlp/language-embeddings#word2vec).

The two figures below come from that video, by Dr. Timothy Hospedales at Yandex. The first shows the class-vector idea.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-248941026300890606b126a00ae621e38b797bfd%2Fgimg-21a7ab06ebb7.png?alt=media" alt=""><figcaption><p>Image by Dr. Timothy Hospedales, Yandex</p></figcaption></figure>

Once classes live in a vector space, for classification, we can use nearest neighbour or manifold-based labeling propagation, as the second figure shows.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-d51d5cb29e1e4f50aace764bbd683b9d8eaa83df%2Fgimg-82e21102e9eb.png?alt=media" alt=""><figcaption><p>Image by Dr. Timothy Hospedales, Yandex</p></figcaption></figure>

Multiple category vectors? Multilabel zero-shot also in the video.

The same idea works for text. Intent recognition is an essential task for goal-oriented dialogue systems: classifying each user utterance with a label from a predefined set. "Zero-Shot Intent Classification with Siamese Networks" does it [with siamese networks](https://medium.com/data-science/zero-shot-intent-classification-with-siamese-networks-35900471c7fd).

#### GPT3 is ZERO, ONE, FEW

With large language models, the shots move into the prompt: GPT-3 style prompting treats zero-, one-, and few-shot use as how many examples you put in the prompt.

The same notes are in [GPT3](/natural-language-processing-nlp/pretrained-language-models#gpt3), [Large Language Models (LLMs)](/generative-ai/large-language-models-llms), and [Prompt](/generative-ai/prompt).

Writing those prompts is its own skill. [Prompt Engineering Tips & Tricks](https://blog.andrewcantino.com/blog/2021/04/21/prompt-engineering-tips-and-tricks/) is Andrew Cantino on what GPT-3 Prompt Engineering is, why it matters, and some tips and tricks to help you do it well. [Open GPT3 prompt engineering](https://medium.com/swlh/openai-gpt-3-and-prompt-engineering-dcdc2c5fcd29) is "OpenAI GPT-3 and Prompt Engineering", which starts from GPT-3's June 2020 release and its 175 billion parameters, 10x GPT-2, and then digs into prompts as the way to get the model to produce text.

The prompt can also turn the model into a classifier. [Using LLMs as black box classifiers](https://cohenori.medium.com/understanding-scikit-llm-86441a5af370) (May 2023) is the n-shot note for this setting.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Zero shot, one shot, few shot (siamese is one shot). This address no longer opens: <https://blog.floydhub.com/n-shot-learning/>
* with siamese networks. This address no longer opens: <https://towardsdatascience.com/zero-shot-intent-classification-with-siamese-networks-35900471c7fd>


# Model Families

Once the problem shape and the learning setting are known, the remaining choice is which algorithm family to reach for. The page gives a map from task type to algorithm family first, then a course-level reference behind it.

The same notes are in [Meta Learning](/deep-learning/meta-learning).

The map is SAS's "Which machine learning algorithm should I use?", designed primarily for beginner to intermediate data scientists or analysts who want to identify and apply machine learning algorithms to their own problems: [A wonderful introduction into machine learning, and how to choose the right algorithm or family of algorithms for the task at hand.](https://blogs.sas.com/content/subconsciousmusings/2017/04/12/machine-learning-algorithm-use/?utm_source=facebook\&utm_medium=cpc\&utm_campaign=analytics-global\&utm_content=US_interests-conversions)

Behind the map sits the course material: [Stanford cs221](https://stanford.edu/~shervine/teaching/cs-221/) is the CS 221 teaching page.


# Next

Next is Evals. Once the task and the learning setting are named, a framed problem still needs a score, a split, and a baseline before a learner is trusted.


# Introduction

A model is not a result until it has a score, a split, and a baseline. After this chapter the reader can choose a metric, a validation scheme, and a test that would catch a broken dataset, and calibration, explanation, and fairness wait in Responsible AI.

The score comes first. [Evaluation Metrics](/evals/evaluation-metrics) collects the supervised and unsupervised measures, from accuracy and precision/recall through ROC, F1, and perplexity, with a reality check on the metric-learning literature. A score is only as honest as the data under it, so [Datasets Reliability & Correctness](/evals/validation) is about why we shouldn't trust models: dataset ablation, shortcut cues, and behavioral testing that ask whether a dataset actually requires the reasoning you think it does.

Once the score and the data can be trusted, the model has to keep earning them. [Training Strategies](/evals/training-strategies) is the framework for a continuous training strategy, with periodic and performance-based retraining driven by data changes, window size, and what to retrain. [Hyper Parameter Optimization](/evals/hyper-parameter-optimization) is searching hyperparameters with Hyperopt, HyperparameterHunter, and a comparison of optimizers, without losing track of experiments.

The last two pages turn evaluation into something a pipeline runs and a field compares. [Data & Model Tests](/evals/data-and-model-tests) validates data in ML pipelines and unit tests models, from Great Expectations-style data checks to pytest and mocks. [Benchmarking](/evals/benchmarking) collects benchmarks across datasets, algorithms, hardware, cloud providers, and platforms, including NLP and multi-task learning leaderboards and notes on scaling networks.

All of it rests on the split being real. [Data Scientists — Stop Using A Random Seed](https://cohenori.medium.com/data-scientists-stop-using-a-random-seed-fd8f2b83ed50) (April 2022) is why a score needs a real split.


# Evaluation Metrics

A trained model needs a number that says whether it is any good, and accuracy alone is rarely that number. The page starts with supervised measures, from accuracy through precision and recall, F1, ROC and AUC, RMSE, and kappa, then moves to perplexity and to choosing the number of clusters when there are no labels, and ends with a reality check on metric-learning benchmarks.

The same notes are in [Calibration](/responsible-ai/calibration) and [Evaluating Recommender Systems](/ai-product/recommender-systems#evaluating-recommender-systems).

## SUPERVISED

Supervised metrics compare predictions with known labels, so they come first. The order is accuracy, then the measures that split its errors apart, then ROC, RMSE, and kappa.

### Accuracy

The simplest of them is accuracy, the share of predictions that are correct. It is the first metric anyone reports, and the rest of this section is about why it is not enough on its own.

### Precision \ Recall \ ROC \ AUC

Accuracy hides which kind of mistake the model makes, and precision and recall are the two numbers that separate them. The same notes are in [Methods and metrics](/problem-framing/multi-label-classification#methods-and-metrics).

Jason Brownlee's [Performance Measures](http://machinelearningmastery.com/classification-accuracy-is-not-enough-more-performance-measures-you-can-use/) on MachineLearningMastery makes the case: classification accuracy, the number of correct predictions out of all predictions made, is not enough, and he lists more performance measures you can use. The point to keep from it is about the confusion matrix. A balanced confusion matrix is better than one that is either one row of numbers and one of zeros, or a column of numbers and a column of zeros. Therefore an algorithm that outputs a lower classification accuracy but has a better confusion matrix wins.

Precision is the # of positive predictions divided by the total number of positive class values predicted:

$$\text{Precision} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}}$$

Low precision can be thought of as many false positives. Recall looks from the other side: the # of positive predictions divided by the number of positive class values in the test data.

$$\text{Recall (sensitivity)} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}}$$

Low recall can be thought of as many false negatives.

#### F1 Harmonic Mean Score

Precision and recall pull against each other, so choosing between two models needs one number that balances them. F1 is their harmonic mean:

$$\text{F1\_Score} = 2 \* \frac{\text{Precision} \* \text{Recall}}{\text{Precision} + \text{Recall}}$$

F1 helps select a model based on a balance between precision and recall. In a multi-class problem, there are many methods to calculate F1, some are more appropriate for balanced data, others are not. [**The best link yet**](https://simonhessner.de/why-are-precision-recall-and-f1-score-equal-when-using-micro-averaging-in-a-multi-class-problem/) on those methods covers micro, macro, and weighted averaging (macro balanced, micro imbalanced, weighted imbalanced). [**Micro vs macro** ](https://datascience.stackexchange.com/questions/15989/micro-average-vs-macro-average-performance-in-a-multiclass-classification-settin/16001)is Shashank Gupta's question on micro- versus macro-averaged performance in a multiclass setting with three classes and a skewed class distribution. [**Micro vs weighted (not a good link**](https://stats.stackexchange.com/questions/169439/micro-vs-weighted-f1-score)**)** is Franck Dernoncourt's question on what to take into account when choosing between a micro and a weighted F1 score, given that macro gives a sense of effectiveness on small classes. [**What is weighted**](https://stats.stackexchange.com/questions/283961/where-does-sklearns-weighted-f1-score-come-from) is Abhishek Divekar's question on where scikit-learn's weighted F1 comes from: the 'weighted' option of f1\_score and fbeta\_score multiplies the classwise F1-scores by the support, the number of examples in that class. [**Micro is accuracy**](https://stackoverflow.com/questions/37358496/is-f1-micro-the-same-as-accuracy) in multi class, and that question asks whether it is always true after many scikit-learn examples where F1 micro came out the same as accuracy.

The same ideas have older names from diagnostic testing, where the population is split into diseased and disease-free individuals:

* **Accuracy = (1 – Error) = (TP + TN)/(PP + NP) = Pr(C), the probability of a correct classification.**
* **Sensitivity (recall) = TP/(TP + FN) = TP/PP = the ability of the test to detect disease in a population of diseased individuals.**
* **Specificity = TN/(TN + FP) = TN / NP = the ability of the test to correctly rule out the disease in a disease-free population.**

Sensitivity and specificity against ROC and AUC is the question behind [What are ?)](http://machinelearningmastery.com/assessing-comparing-classifier-performance-roc-curves-2/), Jason Brownlee's post on assessing and comparing classifier performance with ROC curves. Accuracy, the percent of correct classifications, is easy to understand and makes comparing classifiers trivial, but it ignores many of the factors that should be taken into account when honestly assessing performance. [**ROC curve and AUC in weka**](https://www.youtube.com/watch?v=j97h_-b0gvw\&list=PLJbE6j2EG1pZnBhOg3_Rb63WLCprtyJag) explains how the curve should look like for the negative or positive predictions, against what is actually plotted.

Averaging F1 across runs raises its own question: mean F1, how do we calculate it? [**How**](https://datascience.stackexchange.com/questions/16179/what-is-the-correct-way-to-compute-mean-f1-score) is Pinkesh Badjatiya's question for ten experiments: average the ten F1 scores, or compute F1 from the average precision and average recall. [**it**](http://rushdishams.blogspot.com/2011/08/micro-and-macro-average-of-precision.html) is the post on micro- and macro-average of precision, recall and F-score, where the F-Score is the equally weighted harmonic mean of the two.

For more than two classes, Boaz Shmueli's Multi-Class Metrics Made Simple series is the walkthrough. [**Multiclass Precision / Recall**](https://shmueli.medium.com/multi-class-metrics-made-simple-part-ii-the-f1-score-ebe8b2c2ca1) is Part II, the F1-score, and [**part 1**](https://shmueli.medium.com/multi-class-metrics-made-simple-part-i-precision-and-recall-9250280bddc2) explains how precision and recall are used and calculated in multi-class classification, starting from a recap of the binary case.

Ranking needs the same measures cut off at the top of the list. [**Precision at K**](https://medium.com/@m_n_malaeb/recall-and-precision-at-k-for-recommender-systems-618483226c54) is Maher Malaeb on how precision and recall, classical metrics for binary classification and document retrieval, were translated to evaluate recommendation systems. The [**formulas, examples**](https://surprise.readthedocs.io/en/latest/FAQ.html#how-to-compute-precision-k-and-recall-k) are in the Surprise documentation FAQ. [**git 1**](https://github.com/scikit-learn/scikit-learn/pull/14859) is the work-in-progress scikit-learn pull request for precision and recall at k by leonardbinet, [**git 2**](https://gist.github.com/mblondel/7337391) is a gist of learning to rank metrics, and [**git 3**](https://github.com/scikit-learn/scikit-learn/issues/7343) is the scikit-learn issue to add precision at K, since average\_precision\_score exists but average precision at k does not seem possible. All three are suggestive, written for a recommendation application.

#### RECALL, PRECISION AND F1

The formulas say what the numbers are. The tutorials here are about moving them, mostly by moving the decision threshold. [**Medium on Controling the decision threshold using the probabilities any model gives, code, samples, tutorial**](https://medium.com/data-science/fine-tuning-a-classifier-in-scikit-learn-66e048c21e65) is Kevin Arvai's fine-tuning of a classifier in scikit-learn: many problems benefit from either precision or recall as the optimal metric, but implementing that takes a detailed process, and his first attempts to fine-tune for recall (sensitivity) were difficult. [**Another good medium explanation on precision / recall / fpr/ tpr etc**](https://medium.com/data-science/beyond-accuracy-precision-and-recall-3da06bea9f6c) is Will Koehrsen's Beyond Accuracy: Precision and Recall, which opens with a model that identifies terrorists boarding flights with greater than 99% accuracy by labeling every passenger as not a terrorist. A third tutorial, Scikit lego on choosing the threshold using grid search, used to sit here; that address no longer opens and is kept at the end of the page. [**Best explanation ever** ](https://www.quora.com/What-is-the-best-way-to-understand-the-terms-precision-and-recall)is the Quora question on the best way to understand the terms "precision" and "recall", with an answer from a Ph.D in Computational Linguistics & Machine Learning, University of Lethbridge (2013).

The intuition that sticks is a memory test. For **recall**, one day, your girlfriend asks you: ‘Sweetie, do you remember all birthday surprises from me?’ This simple question makes your life in danger. To extend your life, you need to recall all 10 surprising events from your memory. So, recall is the ratio of a number of events you can correctly recall to a number of all correct events. If you can recall all 10 events correctly, then, your recall ratio is 1.0 (100%). If you can recall 7 events correctly, your recall ratio is 0.7 (70%).

**Precision** is the other half of the same story. For example, you answers 15 times, 10 events are correct and 5 events are wrong. This means you can recall all events but it’s not so precise. So, precision is the ratio of a number of events you can correctly recall to a number all events you recall (mix of correct and wrong recalls). In other words, it is how precise of your recall. From the previous example (10 real events, 15 answers: 10 correct answers, 5 wrong answers), you get 100% recall but your precision is only 66.67% (10 / 15).

The same two ratios can be read straight off the table. [**Confusion matrix wise**](http://www.kdnuggets.com/faq/precision-recall.html): bottom line is recall (% correct out of positive cases), right column is precision (% of POS predictions) & % accuracy in diagonal. The figure shows that layout.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-09e5199f8c7ffb867d669be98bb00f264a77a82c%2Fgimg-ad16f58d3e42.png?alt=media" alt=""><figcaption><p>Confusion matrix and precision/recall layout.</p></figcaption></figure>

The figure is copied from the original hosted image. Back to one number: the [**F1 score**](http://machinelearningmastery.com/classification-accuracy-is-not-enough-more-performance-measures-you-can-use/), from the same performance-measures post, conveys the balance between the precision and the recall:

$$2\*((precision\*recall)/(precision+recall))$$

Yet another(pretty good) source, and another (bad) source for explaining precision, recall, accuracy, true positive rate etc., used to sit here. Both addresses no longer open and are kept at the end of the page, and an archived copy of the first is further down.

How to use precision and recall comes down to the threshold, and the answer by aurelien geron walks through it:

* **In a binary classifier, the decision function is the function that produces a score for the positive class.**
* **In a logistic regression classifier, that decision function is simply a linear combination of the input features.**
* **If that score is greater than some threshold that you choose, then the classifier "predicts" the positive class, or else it predicts the negative class.**
* **If you want your model to have high precision (at the cost of a low recall), then you must set the threshold pretty high. This way, the model will only predict the positive class when it is absolutely certain. For example, you may want this if the classifier is selecting videos that are safe for kids: it's better to err on the safe side.**
* **Conversely, if you want high recall (at the cost of a low precision) then you must use a low threshold. For example, if the classifier is used to detect intruders in a nuclear plant, then you probably want to detect all actual intruders, even if it means getting a lot of false alarms (called "false positives").**
* **If you make a few assumptions about the distribution of the data (i.e., the positive and negative class are separated by a linear boundary plus Gaussian noise), then computing the logistic of the score gives you the probability that the instance belongs to the positive class. A score of 0 corresponds to the 50% probability. So by default, a LogisticClassifier predicts the positive class if it estimates the probability to be greater than 50%. In general, this sounds like a reasonable default threshold, but really it all depends on what you want to do with the classifier.**
* **If the assumptions I mentioned above were perfect, then if the Logistic Classifier outputs a probability of X% for an instance, it means there is exactly X% chance that it's positive. But in practice, the assumptions are imperfect, so I try to always make it clear that we are talking about an "estimated probability", not an actual probability.**

#### ROC CURVES

Sweeping that threshold draws a curve, and the question becomes which curve to read. [**Diff between precision recall to roc curve**](https://www.quora.com/What-is-the-difference-between-a-ROC-curve-and-a-precision-recall-curve-When-should-I-use-each) is the comparison of the two. The rule of thumb is in "What is ROC AUC and PR AUC and when to use then (i.e for imbalanced data use PRAUC)"; its original address no longer opens, and an archived copy is further down. [**What is AUC (AUROC)**](https://stats.stackexchange.com/questions/132777/what-does-auc-stand-for-and-what-is-it) is the question from someone who searched high and low for what AUC, as related to prediction, stands for and means.

Regression has no threshold to move, so its errors are measured directly. [**(RMSE - what is?)**](https://stats.stackexchange.com/questions/56302/what-are-good-rmse-values): it is important to recall that RMSE has the same unit as the dependent variable (DV). It means that there is no absolute good or bad threshold, however you can define it based on your DV. For a datum which ranges from 0 to 1000, an RMSE of 0.7 is small, but if the range goes from 0 to 1, it is not that small anymore. However, although the smaller the RMSE, the better, what counts as small depends on that range.

[**(R^2 vs RMSE)**](https://stats.stackexchange.com/questions/142248/difference-between-r-square-and-rmse-in-linear-regression): R-squared is conveniently scaled between 0 and 1, whereas RMSE is not scaled to any particular values. This can be good or bad; obviously R-squared can be more easily interpreted, but with RMSE we explicitly know how much our predictions deviate, on average, from the actual values in the dataset. So in a way, RMSE tells you more. I also found this [**video**](https://www.youtube.com/watch?v=aq8VU5KLmkY) really helpful: zedstatistics' Introduction to REGRESSION!, on SSE, SSR, SST, R-squared, and errors (ε vs. e).

Back on classifiers, one more measure corrects for class balance. [**Kappa**](https://stats.stackexchange.com/questions/82162/cohens-kappa-in-plain-english) measures accuracy while considering imbalanced datasets; the question is Cohen's kappa in plain English, from a reader of a data mining book who could not see how kappa evaluates the prediction performance of classifiers.

**References:** [**A Survey on Deep Learning in Medical Image Analysis**](https://arxiv.org/abs/1702.05747) reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, surveying deep learning for image classification, object detection, and more.

### Perplexity

Classification scores come from a loss, and so does the main score for language models. The same notes are in [LOSS](/deep-learning/deep-neural-nets#loss), [LOSS IN KERAS](/deep-learning/deep-neural-frameworks#loss-in-keras), and [Metrics](/generative-ai/large-language-models-llms#metrics).

[perplexity and accuracy in classification](https://medium.com/unpackai/perplexity-and-accuracy-in-classification-114b57bd820d) starts from perplexity as a metric used essentially for language models. Since it is defined as the exponential of the model's cross entropy, the piece asks what perplexity can mean for classification, the basic case that also uses cross entropy as a loss.

### UNSUPERVISED

Without labels, the score is about structure, and the first question for k-means is how many clusters to ask for. The same notes are in [Clustering Algorithms](/predictive-ml/clustering-algorithms).

The comparison to read is Silhouette Analysis vs Elbow Method vs Davies-Bouldin Index: Selecting the optimal number of clusters for KMeans clustering. Its original address no longer opens, and an archived copy is in the next section.

##

The notes kept after this separator are mostly the same sources as above, at other addresses. The Towards Data Science address of Multiclass Precision / Recall, <https://towardsdatascience.com/multi-class-metrics-made-simple-part-ii-the-f1-score-ebe8b2c2ca1>, now opens Vinicius Trevisan's multiclass classification evaluation with ROC curves and ROC AUC, which adapts the most used classification evaluation metric to the multiclass classification problem with OvR and OvO strategies. The part 1 address on the same site, <https://towardsdatascience.com/multi-class-metrics-made-simple-part-i-precision-and-recall-9250280bddc2>, opens Towards Data Science, the publication that calls itself a home for data science and AI. Another good medium explanation on precision / recall / fpr/ tpr etc. is also at <https://towardsdatascience.com/beyond-accuracy-precision-and-recall-3da06bea9f6c>.

Yet another(pretty good) source survives as an archived copy, <https://web.archive.org/web/2020/http://blog.exsilio.com/all/accuracy-precision-recall-f1-score-interpretation-of-performance-measures/>, the Exsilio Blog post on accuracy, precision, recall and F1 score as an interpretation of performance measures: how to evaluate the performance of a model in Azure ML and understanding “Confusion Metrics”. What is ROC AUC and PR AUC and when to use then (i.e for imbalanced data use PRAUC) is archived at <https://web.archive.org/web/2020/http://www.chioka.in/differences-between-roc-auc-and-pr-auc/>, and Silhouette Analysis vs Elbow Method vs Davies-Bouldin Index: Selecting the optimal number of clusters for KMeans clustering is archived at <https://web.archive.org/web/2020/https://gdcoder.com/silhouette-analysis-vs-elbow-method-vs-davies-bouldin-index-selecting-the-optimal-number-of-clusters-for-kmeans-clustering/>.

## A metric learning reality check

Every metric above assumes the benchmark it is reported on is fair, and metric learning is where that assumption was checked. The same notes are in [A Reality Check](/natural-language-processing-nlp/a-reality-check) and [VECTOR SIMILARITY SEARCH](/deep-learning/representations#vector-similarity-search).

The [**Medium**](https://medium.com/@tkm45/updates-to-a-metric-learning-reality-check-730b6914dfe7) post is the updates to "A Metric Learning Reality Check". The [**Git**](https://github.com/KevinMusgrave/pytorch-metric-learning) repo is pytorch-metric-learning, billed as the easiest way to use deep metric learning in your application: modular, flexible, and extensible, written in PyTorch. The [**Website**](https://kevinmusgrave.github.io/powerful-benchmarker/papers/mlrc/) is the reality-check paper's page on Powerful Benchmarker.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Medium on Controling the decision threshold using the probabilities any model gives, code, samples, tutorial. This address no longer opens: <https://towardsdatascience.com/fine-tuning-a-classifier-in-scikit-learn-66e048c21e65>
* Scikit lego on choosing the threshold using grid search. This address no longer opens: <https://scikit-lego.readthedocs.io/en/latest/meta.html#Thresholder>
* Yet another(pretty good) source. This address no longer opens: <http://blog.exsilio.com/all/accuracy-precision-recall-f1-score-interpretation-of-performance-measures/>
* Another (bad) source **for explaining, precision, recall, accuracy, true positive rate etc.** This address no longer opens: <https://chrisalbon.com/machine-learning/precision\\_recall\\_and\\_F1\\_scores.html>
* What is ROC AUC and PR AUC and when to use then (i.e for imbalanced data use PRAUC). This address no longer opens: <http://www.chioka.in/differences-between-roc-auc-and-pr-auc/>
* Silhouette Analysis vs Elbow Method vs Davies-Bouldin Index: Selecting the optimal number of clusters for KMeans clustering. This address no longer opens: <https://gdcoder.com/silhouette-analysis-vs-elbow-method-vs-davies-bouldin-index-selecting-the-optimal-number-of-clusters-for-kmeans-clustering/>


# Datasets Reliability & Correctness

Or why we shouldn't trust models

A high score only means something if the dataset actually requires the reasoning you think it does, so this page is about not over-trusting models or benchmarks. It starts with shortcut cues and the dataset ablations that expose them, then a debiasing example from fact verification, then behavioral testing with CheckList.

The warning sign is the [Clever Hans effect](https://thegradient.pub/nlps-clever-hans-moment-has-arrived), in relations to cues left in the dataset that models find, instead of actually solving the defined task! The defence is borrowed from model work. Ablating, i.e. removing, part of a model and observing the impact this has on performance is a common method for verifying that the part in question is useful. If performance doesn't go down, then the part is useless and should be removed. Carrying this method over to datasets, it should become common practice to perform dataset ablations, as well, for example:

* Provide only incomplete input (as done in the reviewed paper): This verifies that the complete input is required. If not, the dataset contains cues that allow taking shortcuts.
* Shuffle the input: This verifies the importance of word (or sentence) order. If a bag-of-words/sentences gives similar results, even though the task requires sequential reasoning, then the model has not learned sequential reasoning and the dataset contains cues that allow the model to "solve" the task without it.
* Assign random labels: How much does performance drop if ten percent of instances are relabeled randomly? How much with all random labels? If scores don't change much, the model probably didn't learning anything interesting about the task.
* Randomly replace content words: How much does performance drop if all noun phrases and/or verb phrases are replaced with random noun phrases and verbs? If not much, the dataset may provide unintended non-content cues, such as sentence length or distribution of function words.

The lesson is that datasets need more love: datasets ablation and public beta before a benchmark is trusted, and inter-prediction agreement as a check on what the models are actually keying on.

Fact verification is a worked case of such a cue. Towards Debiasing Fact Verification Models ([Paper](https://arxiv.org/abs/1908.05267)) starts from the claim that fact verification requires validating a claim in the context of evidence, and shows that in the popular FEVER dataset this might not necessarily be the case: claim-only classifiers perform competitively with top evidence-aware models. The paper investigates the cause and identifies strong cues for predicting labels solely based on the claim, without considering any evidence.

Ablation removes pieces; behavioral testing and CHECKLIST go the other way and probe what the model can do. The [Blog](https://amitness.com/2020/07/checklist/) is Amit Chaudhary's overview of the “CheckList” framework for fine-grained evaluation of NLP models. The [Youtube](https://www.youtube.com/watch?v=L3gaWctPg6E) video is Connor Shorten's CheckList Explained! (ACL 2020 Best Paper). The [paper](https://arxiv.org/pdf/2005.04118.pdf) itself is Beyond Accuracy: Behavioral Testing of NLP models with CheckList, and the [git](https://github.com/marcotcr/checklist) repo, marcotcr/checklist, is its code.

The same paper, ע Beyond Accuracy: Behavioral Testing of NLP models with CheckList, was discussed in Hebrew with its abstract page at <https://arxiv.org/abs/2005.04118>:

מאמר מעניין ומאוד יישומי שזכה בbest paper award בACL 2020. [Yonatan hadar on the subject in hebrew](https://www.facebook.com/groups/MDLI1/permalink/1627671704063538/) is that post in the Machine & Deep learning Israel group.


# Training Strategies

A model that scored well once still has to be retrained as the data moves, and the question is when and how. The page answers it with one framework for a continuous training strategy: periodic and performance-based retraining driven by data changes, window size, and what to retrain.

The same notes are in [Drift](/ai-engineering/mlops-monitoring-and-alerts#drift), [Incremental Learning](/decision-intelligence/incremental-learning), and [Online Learning](/problem-framing/online-learning).

The **(amazing) Framework for a successful training strategy** is <https://towardsdatascience.com/framework-for-a-successful-continuous-training-strategy-8c83d17bb9dc>. It lays out the choices in order: periodic, performance based, driven by data changes, dynamic window size, dynamic data selection, what to retrain and the level.


# Hyper Parameter Optimization

Once a model has a metric, the next question is which settings get the best score without losing track of the experiments that produced it. The page starts with Hyperopt and its search algorithms, moves to HyperparameterHunter, which records every run, and ends with a package that compares optimizers.

The same notes are in [Drift](/ai-engineering/mlops-monitoring-and-alerts#drift), [HYPER PARAM GRID SEARCHES](/deep-learning/deep-neural-nets#hyper-param-grid-searches), and [Hyper param optimization](/deep-learning/meta-learning#hyper-param-optimization).

[**Using HyperOpt**](http://hyperopt.github.io/hyperopt/) is the documentation for Hyperopt, Distributed Asynchronous Hyper-parameter Optimization. It offers **Random Search** and **Tree of Parzen Estimators (TPE)**. Hyperopt has been designed to accommodate Bayesian optimization algorithms based on Gaussian processes and regression trees, but these are not currently implemented. All algorithms can be run either serially, or in parallel by communicating via [**MongoDB**](http://www.mongodb.org/).

Search is only half of the job; the other half is keeping the results, which is why this page overlaps with Mlflow, Hyperparameterhunter, hyperopt, concept drift, and unit tests. The overview that tied those together no longer opens and is kept at the end of the page. The same notes are in [ML Experiment Management](/ai-engineering/experiment-management).

For the search itself, [**Hyperopt**](http://hyperopt.github.io/hyperopt/) is the tool for hyperparameter search. [**HyperparameterHunter**](https://github.com/HunterMcGushion/hyperparameter_hunter) adds the record keeping: easy hyperparameter optimization and automatic result saving across machine learning algorithms and libraries. It provides a wrapper for machine learning algorithms that saves all the important data. Simplify the experimentation and hyperparameter tuning process by letting HyperparameterHunter do the hard work of recording, organizing, and learning from your tests, all while using the same libraries you already do. Don't let any of your experiments go to waste, and start doing hyperparameter optimization the way it was meant to be.

The two are not free to swap. An implementation and comparison found HH slower than HO due to usage of skopt; that write-up is also kept at the end of the page. To compare more than two, [**HumpDay**](https://github.com/microprediction/humpday) is a package that compares optimization algorithms and ranks them.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Mlflow, Hyperparameterhunter, hyperopt, concept drift, unit tests. This address no longer opens: <https://towardsdatascience.com/putting-ml-in-production-ii-logging-and-monitoring-algorithms-91f174044e4e>
* Implementation and comparison **- HH slower than HO due to usage of skopt.** This address no longer opens: <https://towardsdatascience.com/putting-ml-in-production-ii-logging-and-monitoring-algorithms-91f174044e4e>


# Data & Model Tests

A score is only trustworthy if the data feeding it and the code producing it are checked every time the pipeline runs. The page starts with data testing, Great Expectations-style checks and data profiling, and then moves to model testing with unit tests, pytest, and mocks.

The same notes are in [Lakes and warehouses](/data/engineering-intro/lakes-and-warehouses#lakes-and-warehouses) and [Pandas](/data/engineering-intro/data-science-tools#pandas).

## Data Testing

The data comes first, because a broken table breaks every model trained on it. The same notes are in [Data Quality](/data/engineering-intro/data-quality) and [Data Validation](/data/engineering-intro/data-quality#data-validation).

[Great expectations](https://greatexpectations.io/) is GX Core, an open source framework for testing, validating, and documenting data quality across modern data pipelines, workflows, and teams. The GitHub [article](https://github.blog/2020-10-01-keeping-your-data-pipelines-healthy-with-the-great-expectations-github-action/) by Hamel Husain, the second in a series on using GitHub for MLOps and data science, is about keeping your data pipelines healthy with the Great Expectations GitHub Action, the point where those checks join continuous integration. The [Youtube](https://www.youtube.com/watch?v=uM9DB2ca8T8) video is the Great Expectations 101 getting-started webinar, and it is the “TDDA” for Unit tests and CI note on this page. Before writing expectations, it helps to know what is in the table: the [DataProfiler git](https://github.com/capitalone/DataProfiler) repo from capitalone asks "What's in your data?" and extracts schema, statistics and entities from datasets.

## Model Testing

Once the data is checked, the code that trains and serves the model needs the same discipline. The same notes are in [Continuous Integration](/ai-engineering/full-stack-and-ops-1/continuous-integration).

The starting point is A great :P [unit test and logging](https://cohenori.medium.com/unit-testing-and-logging-for-data-science-d7fb8fd5d217) post on medium - it's actually mine :) After that, a mind blowing [lecture](https://www.youtube.com/watch?v=1fHGXOfiDO0\&feature=youtu.be) about unit testing your data using Voluptuous & engrade & TDDA lecture ties the model tests back to the data tests above.

The Python tooling follows. [Unit tests in python](https://jeffknupp.com/blog/2013/12/09/improve-your-python-understanding-unit-testing/) is the written introduction. [Unit tests in python - youtube](https://www.youtube.com/watch?v=6tNS--WetLI) is Corey Schafer's Python Tutorial: Unit Testing Your Code with the unittest Module. [Unit tests asserts](https://docs.python.org/3/library/unittest.html#unittest.TestCase.debug) is the unittest documentation, the unit testing framework whose list of assert methods is the part to skip to if you already know the basic concepts of testing. [Auger - automatic unit tests, has a blog post inside](https://github.com/laffra/auger) is Automated Unittest Generation for Python, but it doesn't work with py 3+.

For data science specifically, [A rather naive unit tests article aimed for DS](https://medium.com/@danielhen/unit-tests-for-data-science-the-main-use-cases-1928d9e7a4d4) is Daniel Hen on when we should use unit testing for data science: the question is not if a DS pipeline needs unit tests, but what to look out for when writing them. A good pytest [tutorial](https://www.tutorialspoint.com/pytest/index.htm) covers Pytest, a testing framework for Python that makes it easy to write simple and scalable test cases, with fixtures, parameterized tests, and detailed error reporting.

Tests that touch outside services need mocks. [Mock](https://medium.com/@yasufumy/python-mock-basics-674c33de1ced) is Let’s be Friends with Mock in Python, on `unittest.mock` and why a library that talks to Google Spreadsheet should not connect to it in every test. [mock 2](https://medium.com/python-pandemonium/python-mocking-you-are-a-tricksy-beast-6c4a1f8d19b2) is Matt Pease's Python Mocking, You Are A Tricksy Beast: mocking is unintuitive and takes several “Ah, now I get it!” moments, but it is a powerful tool for enabling automated unit tests.


# Benchmarking

A score means little until it is compared with something: another dataset, another algorithm, another machine. The page walks those comparisons in order, from datasets and classifiers through NumPy's BLAS, hardware, cloud providers, and deep-learning platforms, then NLP and multi-task learning leaderboards, and ends with notes on scaling networks.

## Algorithms

The comparisons start with what is being run, the data and the algorithm, before the machine it runs on.

### Datasets

The first benchmark is of the data itself. EFF FF Benchmarks in AI used to be the pointer here; that address no longer opens and is kept at the end of the page. On the library side, [scikit bench](https://github.com/IntelPython/scikit-learn_bench) is described in its own words: "scikit-learn\_bench benchmarks various implementations of machine learning algorithms across data analytics frameworks. It currently support the scikit-learn, DAAL4PY, cuML, and XGBoost frameworks for commonly used machine learning algorithms."

### Algorithms (classifiers)

With the implementations benchmarked, the next question is which classifier to pick. [Comparing](https://martin-thoma.com/comparing-classifiers/) is the side-by-side of accuracy, speed, memory and 2D visualization of classifiers, and every model in it links to its scikit-learn page, each with its own gallery of examples:

[SVM,](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) [k-nearest neighbors,](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html) [Random Forest,](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) [AdaBoost Classifier,](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html) [Gradient Boosting,](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) [Naive, Bayes,](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html) [LDA,](http://scikit-learn.org/0.16/modules/generated/sklearn.lda.LDA.html) [QDA,](http://scikit-learn.org/0.16/modules/generated/sklearn.qda.QDA.html) [RBMs,](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.BernoulliRBM.html) [Logistic Regression,](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) [RBM](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.BernoulliRBM.html) + Logistic Regression Classifier

The SVC, KNeighborsClassifier, RandomForestClassifier, AdaBoostClassifier, GradientBoostingClassifier, GaussianNB, and LogisticRegression pages all list the classifier comparison among their examples, LDA and QDA point at the scikit-learn 0.16.1 documentation, and BernoulliRBM shows Restricted Boltzmann Machine features for digit classification. The same comparison holds inside one model family: [LSTM vs cuDNN LSTM](https://chainer.org/general/2017/03/15/Performance-of-LSTM-Using-CuDNN-v5.html) shows that batch size of power 2 matters, and the latter is faster.

### Numpy Blas

Under every classifier sits linear algebra, so the library NumPy links against changes the timing. [How do i know which version of blas is installed](https://stackoverflow.com/questions/37184618/find-out-if-which-blas-library-is-used-by-numpy) is the question of how to be sure NumPy uses a BLAS library at all when it was installed from a package manager rather than compiled by hand. [Benchmark OpenBLAS, Intel MKL vs ATLAS](https://github.com/tmolteno/necpp/issues/18) is a shared benchmark of LAPACK / BLAS libraries from someone replacing CPU and math library for a huge simulation model, who concluded that Intel MKL is the best. The figure is that benchmark.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-451321a9994243937754c8d102cc407ed14a875b%2Fgimg-eaf5612396a1.png?alt=media" alt=""><figcaption><p>OpenBLAS, Intel MKL vs ATLAS benchmark.</p></figcaption></figure>

Another comparison of NumPy BLAS builds is in the next figure; its source post no longer opens and is kept at the end of the page. Both figures are copied from the original hosted images.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-c727009c30f652d201afbb682e34102e068a7af5%2Fgimg-02e0dbef7ebf.png?alt=media" alt=""><figcaption><p>Another NumPy BLAS comparison.</p></figcaption></figure>

### Hardware

Past the libraries, the machine decides the rest, and the first choice is GPU versus CPU. [Nvidia](https://www.phoronix.com/scan.php?page=article\&item=nvidia-rtx2080ti-tensorflow\&num=1) compares the 1070 vs 1080 vs 2080. [Cpu vs GPU benchmarking for CNN/Test/LTSM/BDLTSM](http://minimaxir.com/2017/07/cpu-or-gpu/) benchmarks TensorFlow on cloud CPUs and finds them cheaper for deep learning than cloud GPUs, because of the massive cost differential afforded by preemptible instances. [Nvidia GPUs](https://www.pugetsystems.com/labs/hpc/TitanXp-vs-GTX1080Ti-for-Machine-Learning-937/) is TitanXp vs GTX1080Ti for Machine Learning: the GTX1080Ti proved as good as the Titan X Pascal at a much lower price, and the new Titan Xp is tested to see how much faster it is. As of March/17, [Nvidia GPUs for desktop](https://medium.com/@timcamber/deep-learning-pc-build-5cffa71ad97) compares cards in terms of price and cuda units, and the bottom line is 1060-1080. [Another bench up to 2013](http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/) is regarding many GPUS vs CPUs in terms of BW.

### Cloud providers

Renting the hardware brings its own comparison, the hardware benchmarks across cloud providers from the gensim series. [Part 1](https://rare-technologies.com/machine-learning-hardware-benchmarks/) and [part 2 y gensim](https://rare-technologies.com/machine-learning-benchmarks-hardware-providers-gpu-part-2/) were that series; both addresses now open PII Tools, a product for analyzing personal data and sensitive information at scale, not the benchmarks.

### Platforms

On a given machine, the framework is the last variable. Cntk vs tensorflow was one comparison; its address no longer opens and is kept at the end of the page. [CNTK, TEnsor, torch, etc on cpu and gpu](https://arxiv.org/pdf/1608.07249.pdf) is the paper Benchmarking State-of-the-Art Deep Learning Software Tools.

### NLP

Leaderboards move the comparison from speed to capability, and language has its own. [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://github.com/google-research/xtreme/blob/master/README.md) is a benchmark for the cross-lingual generalization ability of pre-trained multilingual models that covers 40 typologically diverse languages and includes nine tasks.

#### Multi-Task Learning

A multi-task benchmark raises the question of how one model should balance the tasks, through loss weighting and shared representations. [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://arxiv.org/abs/1705.07115) (Yarin Gal), with an unofficial implementation on [GitHub](https://github.com/ranandalon/mtl), states the problem: "In this paper we make the observation that the performance of such systems is strongly dependent on the relative weighting between each task’s loss. Tuning these weights by hand is a difficult and expensive process, making multi-task learning prohibitive in practice. We propose a principled approach to multi-task deep learning which weighs multiple loss functions by considering the homoscedastic uncertainty of each task. " [Ruder on Multi Task Learning](https://ruder.io/multi-task/) gives the reason to share at all: "By sharing representations between related tasks, we can enable our model to generalize better on our original task. This approach is called Multi-Task Learning (MTL) and will be the topic of this blog post."

### GLUE

For English understanding, the standard leaderboard is GLUE and its successor. The same notes are in [BERT](/natural-language-processing-nlp/pretrained-language-models#bert).

[Glue / super glue](https://gluebenchmark.com/leaderboard/) is the leaderboard of the General Language Understanding Evaluation (GLUE) benchmark, a collection of resources for training, evaluating, and analyzing natural language understanding systems.

### State of the art in AI

Beyond single leaderboards, state of the art can be tracked across the field, in terms of [domain X datasets](https://www.stateoftheart.ai/).

### Scaling networks and predicting performance of NN

The last comparison is forward-looking: predicting train time and accuracy before scaling networks across GPUs. [A great overview of NN types](https://www.youtube.com/watch?v=lgK0BlXdOCw\&feature=youtu.be) is the video for it, but the idea behind the video is to create a system that can predict train time and possibly accuracy when scaling networks using multiple GPUs, there is also a nice slide about general hardware recommendations. The figure below is that slide, copied from the original hosted image.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-763ceb403e1604ae1db522eb7b0e8ad9f483bbe6%2Fgimg-5fcba0363084.png?alt=media" alt=""><figcaption><p>Hardware recommendations when scaling networks across GPUs.</p></figcaption></figure>

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Another comparison (NumPy BLAS). This address no longer opens: <http://markus-beuckelmann.de/blog/boosting-numpy-blas.html>
* EFF FF Benchmarks in AI. This address no longer opens: <https://www.eff.org/ai/metrics>
* Cntk vs tensorflow. This address no longer opens: <http://minimaxir.com/2017/06/keras-cntk/>


# Next

Next is Predictive ML. The score is in place, and the first models are the classical ones. Each classical learner is one inductive bias, and now each one can be measured with the metric, split, and tests from this chapter.


# Introduction

Each classical learner is one inductive bias, and each bias uses the one before it. After this chapter the reader can pick a linear model, a tree, a cluster, a forecast, or a graph method and say what it assumes. The chapter moves from supervised learners with a label, to unsupervised methods without one, to signals that unfold in time, and ends with graphs.

## Supervised

The supervised part starts before there is a label at all. [Data Mining](/predictive-ml/data-mining) is about association rules, Apriori, FP-Growth, and how to read support, confidence, and lift when the goal is frequent itemsets rather than a prediction. The first real boundary between classes comes next: [Linear Separator Algorithms](/predictive-ml/linear-separator-algorithms) covers support vector machines and related linear separators, with kernels, SMO, libsvm vs liblinear, and overfitting advice. The opposite bias, trusting whatever is close, is [Nearest Neighbors](/predictive-ml/nearest-neighbors), the scalable nearest-neighbour libraries and notes on choosing k.

Once a model can fit, it can also fit noise. [Regularization](/predictive-ml/regularization) is the penalty added to a loss so that such a model is discouraged, with norms, L1/L2 sparsity intuition, and priors. [Probabilistic Models](/predictive-ml/probabilistic-models) replaces geometry with probability: naive Bayes, Bayesian networks, Markov models, HMMs, IOHMM, and CRF. When the target is continuous rather than a class, [Regression](/predictive-ml/regression) covers the algorithms, kernel regression, and the error metrics used to judge them.

Trees give a different bias again. [Decision Trees](/predictive-ml/decision-trees) covers Hellinger splits, CART, KD-trees, random forests, and extremely randomized trees, and [Ensembles](/predictive-ml/ensembles) then combines many models through voting, bagging, boosting, and stacking, up to gradient boosting, XGBoost, and CatBoost. The last two pages evolve the learner itself: [Genetic Algorithms & Genetic Programming](/predictive-ml/genetic-algorithms-and-genetic-programming) contrasts evolving parameters with evolving programs, and [Learning Classifier Systems](/predictive-ml/learning-classifier-systems) defines LCS and XCS, where the learner is a rule population rather than a single model.

## Unsupervised

Without a label, the question becomes what structure is already in the data. [Clustering Algorithms](/predictive-ml/clustering-algorithms) groups unlabeled points, from k-means and GMM through density-based and constrained clustering. [Dimensionality Reduction Methods](/predictive-ml/dimensionality-reduction-methods) projects high-dimensional data with PCA, SVD, kernel PCA, LDA, ICA, LSA, and manifold methods such as t-SNE. [Anomaly Detection](/predictive-ml/anomaly-detection) separates novelty from outlier detection and runs from elliptic envelopes and LOF through isolation forests and one-class SVM.

Logs are a common unlabeled source of their own. [Process Mining](/predictive-ml/process-mining) recovers the real process from event logs when the documented one is incomplete or ignored, and [Log Parsing / Templatization](/predictive-ml/templatization) turns raw log lines into templates and uses them for log-based anomaly detection.

## Signals

Some data has an order in time, and that order is the signal. [Forecasting](/predictive-ml/forecasting) covers time-series components, stationarity, decomposition, forecasting methods, and accuracy. [Time Series Search](/predictive-ml/time-series-search) compares series with DTW, then clusters them and detects anomalies in them. [Fourier Transform](/predictive-ml/fourier-transform) gives the frequency-domain view of a signal, plus wavelets, before modeling, and [Digital Signal Processing (DSP)](/predictive-ml/digital-signal-processing-dsp) is the toolbox shelf of SciPy, Librosa, and beat-detection helpers.

Audio is the signal case with its own path. [Audio Basics](/predictive-ml/audio-basics) is Ketan Doshi's series from sound and spectrograms to speech recognition, [Audio Terminology](/predictive-ml/audio-terminology) defines source separation and sound event detection so the task names match the methods, [Audio Feature Engineering](/predictive-ml/audio-feature-engineering) covers Mel spectrograms, MFCC, and related features, and [Audio Algorithms](/predictive-ml/audio-algorithms) collects the concrete methods and packages for those tasks.

## Graphs

The last bias is that the relations between points matter more than the points. [Graph Theory](/predictive-ml/graph-theory) covers centrality, community detection, courses, and tools, and [Social Network Analysis](/predictive-ml/social-network-analysis) is its applied social-graph companion, with definitions, people, and traits that spread in networks.


# Data Mining

Before any label exists, a transaction table can already say which items tend to show up together. This page answers that with association rules: first the slides, terms, and textbook chapter that define the problem, then Apriori and its Python implementations, then FP-Growth, and finally how to judge a rule with support, confidence, and lift.

### ASSOCIATION RULES

Association analysis starts from market basket data, where each row is one transaction. The same notes are in [Log Parsing / Templatization](/predictive-ml/templatization).

The [Association rules slides](https://www.slideshare.net/wanaezwani/apriori-and-eclat-algorithm-in-association-rule-mining) walk through the Apriori and Eclat algorithms in association rule mining. The [Terms](https://www.kdnuggets.com/2016/04/association-rules-apriori-algorithm-tutorial.html) are the KDnuggets tutorial on association rules and the Apriori algorithm. The [Paper — basic concepts and algo](https://www-users.cs.umn.edu/~kumar001/dmbook/ch5_association_analysis.pdf) is the textbook chapter on association analysis, which starts from the customer purchase data that grocery checkouts collect every day and the market basket transactions table built from it.

A four-part Knoldus series used to follow the same arc, and its original blog links are in Deprecated links. In order, it covered:

1. Apriori
2. Association rules
3. Fp-growth
4. Fp-tree construction

**APRIORI**

Apriori is the first algorithm to learn, because it finds frequent item sets by growing them one item at a time. The [Apyori tut](https://stackabuse.com/association-rule-mining-via-apriori-algorithm-in-python/) frames association rule mining as a technique to identify underlying relations between different items, using a supermarket as the example, and the matching library is [git](https://github.com/ymoch/apyori), a simple implementation of Apriori in Python. When that is too slow, [Efficient apriori](https://github.com/tommyod/Efficient-Apriori) is an efficient Python implementation of the same algorithm.

For the idea behind it, MachineLearningMastery's market basket analysis with association rule learning covers [One of the best known association rules algorithm](https://machinelearningmastery.com/market-basket-analysis-with-association-rule-learning/). [A very good visual example of a transaction DB with the apriori algorithm step by step](http://www.lessons2all.com/Apriori.php) is Vishwanath Pai's Lessons2all example of how to find a frequent item set in a transaction database, with a detailed solution. A Python 3.0 code walkthrough used to sit here; that address now points elsewhere and is kept at the end of the page.

In practice, [Mlxtnd](http://rasbt.github.io/mlxtend/api_subpackages/mlxtend.frequent_patterns/) is Sebastian Raschka's mlxtend frequent patterns module, and the GeeksforGeeks [tutorial](https://www.geeksforgeeks.org/implementing-apriori-algorithm-in-python/) implements Apriori in Python. The mlxtend module exposes four pieces:

1. Apriori
2. Rules
3. pgrowth
4. fpmax

**FP Growth**

Apriori rescans the data for every candidate set, so FP-Growth builds a tree once and grows patterns from it. Well Academy's video on data mining FP-growth shows [How to construct the fp-tree](https://www.youtube.com/watch?v=gq6nKbye648) with an FP-tree example. The rest of the author's list of sources:

2. The same example, but with a graph that shows that lower support cost less for fp-growth in terms of calc time.
3. Coursera video.
4. Another clip video
5. [How to validate these algorithms](https://stackoverflow.com/questions/32843093/how-to-validate-association-rules) — probably the best way is confidence/support/lift

Validation comes back to those three measures. It depends on your task. But usually you want all [three to be high.](https://stats.stackexchange.com/questions/229523/association-rules-support-confidence-and-lift)

* high support: should apply to a large amount of cases
* high confidence: should be correct often
* high lift: indicates it is not just a coincidence

Once both algorithms are clear, the last question is whether they give different answers:

1. [Difference between apriori and fp-growth](https://www.quora.com/What-is-the-difference-between-FPgrowth-and-Apriori-algorithms-in-terms-of-results) explains that both are frequent pattern mining algorithms, but they differ in how they represent, generate, and output frequent itemsets, which leads to practical differences in results, performance, and usability.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Apriori. This address no longer opens: <https://blog.knoldus.com/machinex-why-no-one-uses-apriori-algorithm-for-association-rule-learning/>
* Association rules. This address no longer opens: <https://blog.knoldus.com/machinex-two-parts-of-association-rule-learning/>
* Fp-growth. This address no longer opens: <https://blog.knoldus.com/machinex-frequent-itemset-generation-with-the-fp-growth-algorithm/>
* Fp-tree construction. This address no longer opens: <https://blog.knoldus.com/machinex-understanding-fp-tree-construction/>
* Coursera video. This address no longer opens: <https://www.coursera.org/learn/data-patterns/lecture/ugqCs/2-5-fpgrowth-a-pattern-growth-approach>
* Python 3.0 code. This address now points to an unrelated site: <http://adataanalyst.com/machine-learning/apriori-algorithm-python-3-0/>


# Linear Separator Algorithms

A support vector machine answers one question: where is the best straight boundary between two classes, and what if the classes are not straight at all? The page starts with the SVM definition and margins, then kernels and the RBF rule of thumb, then how SMO, libsvm, and liblinear actually train it, and then the multi-class, regression, and clustering variants. It ends with regularization, grid search, and overfitting advice for C and gamma.

### SUPPORT VECTOR MACHINES (SVM)

The starting point is the plain two-class SVM and its margin. The same notes are in [ONE CLASS SVM](/predictive-ml/anomaly-detection#one-class-svm).

The OpenCV [Definition](http://docs.opencv.org/3.0-beta/modules/ml/doc/support_vector_machines.html) explains that SVM was originally a technique for building an optimal binary (2-class) classifier, later extended to regression and clustering, and that it is a partial case of kernel-based methods. Jake VanderPlas's Python Data Science Handbook chapter, In-Depth: Support Vector Machines, is the hands-on [tutorial](https://jakevdp.github.io/PythonDataScienceHandbook/05.07-support-vector-machines.html).

In the author's notes, SVM is built for an optimal 2-class classifier and extended for regression and clustering problems (1 class). It is kernel-based: it maps feature vectors into a higher-dimensional space using a kernel function, then builds an optimal linear discriminating function in this space (linear?) or an optimal hyper-plane (RBF?) that fits the training data. In case of SVM, the kernel is not defined explicitly, but a distance needs to be defined between any 2 points in the hyper-space.

The solution is optimal, the margin is maximal, between the separating hyper-plane and the nearest feature vectors. The feature vectors that are the closest to the hyper-plane are called support vectors, which means that the position of other vectors does not affect the hyper-plane (the decision function). The model produced by support vector classification (as described above) depends only on a subset of the training data, because the cost function for building the model does not care about training points that lie beyond the margin.

### KERNELS

That definition leaned on a kernel without saying what one is, so the next step is the kernel trick itself. [What are kernels in SVM](https://www.quora.com/What-are-Kernels-in-Machine-Learning-and-SVM) - intuition and example - is the Quora thread on what kernels are in machine learning and SVM and why we need them.

A kernel allows us to do certain calculations faster which otherwise would involve computations in higher dimensional space. K(x, y) = \<f(x), f(y)>. Here K is the kernel function, x, y are n dimensional inputs. f is a map from n-dimension to m-dimension space. < x,y> denotes the dot product. usually m is much larger than n. Normally calculating \<f(x), f(y)> requires us to calculate f(x), f(y) first, and then do the dot product. These two computation steps can be quite expensive as they involve manipulations in m dimensional space, where m can be a large number. Result is ONLY a scalar, i..e., 1-dim space. We don’t need to do that calc if we use a clever kernel.

Example:

Simple Example: x = (x1, x2, x3); y = (y1, y2, y3). Then for the function f(x) = (x1x1, x1x2, x1x3, x2x1, x2x2, x2x3, x3x1, x3x2, x3x3), the kernel is K(x, y ) = (\<x, y>)^2.

Let's plug in some numbers to make this more intuitive: suppose x = (1, 2, 3); y = (4, 5, 6). Then:

f(x) = (1, 2, 3, 2, 4, 6, 3, 6, 9) and f(y) = (16, 20, 24, 20, 25, 30, 24, 30, 36)

\*\*\<f(x), f(y)> = 16 + 40 + 72 + 40 + 100+ 180 + 72 + 180 + 324 = 1024 i.e., 1\*16+2\*20+\*3\*24..

A lot of algebra. Mainly because f is a mapping from 3-dimensional to 9 dimensional space.

With a kernel its faster.

K(x, y) = (4 + 10 + 18 ) ^2 = 32^2 = 1024

A kernel is a magical shortcut to calculate even infinite dimensions!

The same thread then ties the shortcut back to the classifier, under [Relation to SVM](https://www.quora.com/What-are-Kernels-in-Machine-Learning-and-SVM)?. The idea of SVM is that y = w phi(x) +b, where w is the weight, phi is the feature vector, and b is the bias. If y> 0, then we classify datum to class 1, else to class 0. We want to find a set of weight and bias such that the margin is maximized.

Previous answers mention that kernel makes data linearly separable for SVM. I think a more precise way to put this is, kernels do not make the the data linearly separable. The feature vector phi(x) makes the data linearly separable. Kernel is to make the calculation process faster and easier, especially when the feature vector phi is of very high dimension (for example, x1, x2, x3,..., x\_D^n, x1^2, x2^2,...., x\_D^2).

That is also why it can be understood as a measure of similarity: if we put the definition of kernel above, \<f(x), f(y)>, in the context of SVM and feature vectors, it becomes \<phi(x), phi(y)>. The inner product means the projection of phi(x) onto phi(y). or colloquially, how much overlap do x and y have in their feature space. In other words, how similar they are.

In a library, the choice comes down to a short parameter list. The OpenCV page lists these [Kernels](http://docs.opencv.org/3.0-beta/modules/ml/doc/support_vector_machines.html):

* SVM::LINEAR Linear kernel. No mapping is done, linear discrimination (or regression) is done in the original feature space. It is the fastest option. $$K(x\_i, x\_j) = x\_i^T x\_j$$.
* SVM::RBF Radial basis function (RBF), a good choice in most cases. $$K(x\_i, x\_j) = e^{-\gamma ||x\_i - x\_j||^2}, \gamma > 0$$.

### [RBF kernel](https://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf) use cases

RBF is the default, but the LIBSVM practical guide, written because beginners often get unsatisfactory results by missing easy but significant steps, says when it behaves no better than a linear kernel. The answer depends on the ratio of instances to features.

When the number of instances << number of features, i.e. 38 instances over 7000 features, RBF=LINEAR: when the number of features is large, we may not need to use RBF over Linear and vice versa (After finding C and gamma).

When the number of instances & features is VERY LARGE, i.e. 20K samples X 20K features, libsvm and liblinear give similar performance, and liblinear is faster by 150 times. Rule of thumb is to use for document classification.

When the number of instances >> number of features, the usual answer is high dimensional mapping using non linear kernel. If we insist on liblinear, -s 2 leads to faster training.

The same trade-off applies one level up, to deep learning. Kdnuggets: When to use DL over SVM and other algorithms. Computationally expensive for a very small boost in accuracy. The article is <http://www.kdnuggets.com/2016/04/deep-learning-vs-svm-random-forest.html>

### SEQUENTIAL MINIMAL OPTIMIZATION (SMO)

Knowing which kernel to use still leaves the question of how the SVM is trained at all. [What is the SMO (SVM) classifier?](https://www.microsoft.com/en-us/research/publication/sequential-minimal-optimization-a-fast-algorithm-for-training-support-vector-machines/?from=http%3A%2F%2Fresearch.microsoft.com%2Fpubs%2F69644%2Ftr-98-14.pdf) - Sequential Minimal Optimization, or SMO. Training a support vector machine requires the solution of a very large quadratic programming (QP) optimization problem. SMO breaks this large QP problem into a series of the smallest possible QP problems. These small QP problems are solved analytically, which avoids using a time-consuming numerical QP optimization as an inner loop. The amount of memory required for SMO is linear in the training set size, which allows SMO to handle very large training sets. Because matrix computation is avoided, SMO scales somewhere between linear and quadratic in the training set size for various test problems, while the standard chunking SVM algorithm scales somewhere between linear and cubic in the training set size. SMO’s computation time is dominated by SVM evaluation, hence SMO is fastest for linear SVMs and sparse data sets. On real-world sparse data sets, SMO can be more than 1000 times faster than the chunking algorithm.

Two libraries implement this family of solvers, and two Stack Overflow questions compare them: [Differences between libsvm and liblinear](https://stackoverflow.com/questions/11508788/whats-the-difference-between-libsvm-and-liblinear) asks how the differences make liblinear faster than libsvm, & [smo vs libsvm](https://stackoverflow.com/questions/23674411/weka-smo-vs-libsvm) asks whether WEKA's SMO is different from LIBSVM, which itself implements an SMO-type algorithm.

### [LibSVM vs LibLinear](https://stackoverflow.com/questions/11508788/whats-the-difference-between-libsvm-and-liblinear)

The first of those threads is worth unpacking, because it decides which library survives a large dataset. libsvm works by using many kernel transforms to turn a non-linear problem into a linear problem beforehand.

From the link above, it seems like liblinear is very much the same thing, without those kernel transforms. So, as they say, in cases where the kernel transforms are not needed (they mention document classification), it will be faster.

The complexity comparison is a short list:

* libsvm (SMO) implementation
* kernel (n^2)
* Linear SVM (n^3)
* liblinear - optimized to deal with linear classification without kernels
* Complexity O(n)
* does not support kernel SVMs.
* Scores higher

n is the number of samples in the training dataset.

Conclusion: In practice libsvm becomes painfully slow at 10k samples. Hence for medium to large scale datasets use liblinear and forget about libsvm (or maybe have a look at approximate kernel SVM solvers such as [LaSVM](http://leon.bottou.org/projects/lasvm), which saves training time and memory usage for large scale datasets).

### [MULTI CLASS SVM](https://www.csie.ntu.edu.tw/~cjlin/papers/multisvm.pdf)

Everything so far separated two classes, so more classes need a strategy for combining binary machines. The [comparison paper from National Taiwan University](https://www.csie.ntu.edu.tw/~cjlin/papers/multisvm.pdf) covers one against all, one against one, and Direct Acyclic Graph SVM (one against one with DAG). bottom line One Against One in LIBSVM.

Before going further, the parameters need a clear picture. [A few good explanation about SVM, formulas, figures, C, gamma, etc.](https://www.quora.com/What-are-C-and-gamma-with-regards-to-a-support-vector-machine) is the Quora answer on what C and gamma mean for a support vector machine.

Math of SVM on youtube:

{% embed url="<https://www.youtube.com/watch?v=1NxnPkZM9bc>" %}

The second walk through the math is Udacity's Georgia Tech Machine Learning lecture, Distance Between Planes Quiz: [Very good but lengthy and chatty example with make-sense math #2](https://www.youtube.com/watch?v=mU_N3nmv0Go\&list=PLAwxTw4SYaPlkESDcHD-0oqVx5sAIgz7O\&index=4). The author's notes from it:

* Linear - maximize the margin, optimal solution, only a few close points are really needed the others are zeroes by the alphas (alpha says “pay attention to this variable”) in the quadratic programming equation. XtX is a similarity function (pairs of points that relate to each other in output labels and how similar to one another, Xi’s point in the same direction) y1y2 are the labels. Therefore further points are not needed. But the similarity is important here(?)
* Non-linear - e.g. circle inside a circle, needs to map to a higher plane, a measure of similarity as XtX is important. We use this similarity idea to map into a higher plane, but we choose the higher plane for the purpose of a final function that behaves likes a known function, such as (A+B)^2. It turns out that (q1,q2,root(2)q1q2) is engineered with that root(2) thing for the purpose of making the multiplication of X^tY, which turns out to be (X^tY)^2. We can substitute this formula (X^tY)^2 instead of the X^tX in the quadratic equation to do that for us.This is the kernel trick that maps the inner class to one side and the outer circle class to the other and passes a plane in between them.
* Similarity is defined intuitively as all the points in one class vs the other.. I think
* A general kernel K=(X^tY + C)^p is a polynomial kernel that can define the above function and others.
* Quadratic eq with possible kernels including the polynomial.

The figure below is the lecture's picture of that mapping, copied from the original hosted image.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-022975ee8d8ffbcc49cef8c6be586f8a8ca935dc%2Fgimg-15322450a860.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

The lesson the author takes from it is that most importantly the kernel function is our domain knowledge. (?) IMO we should choose a kernel that fits our feature data. The output of K is a number(?), and infinite dimensions are possible as well. The Mercer condition - it acts like a distance\similar so that is the “rule” of which a kernel needs to follow. The [Super good lecture on MIT OPEN COURSE WARE](https://www.youtube.com/watch?v=_PwhiWxHK8o) expands on the quadratic equations that were introduced in the previous course above.

### SUPPORT VECTOR REGRESSION (SVR)

The same margin idea also works when the target is a number instead of a class. The same notes are in [Regression](/predictive-ml/regression) and [REGRESSION ALGORITHMS](/predictive-ml/regression#regression-algorithms).

scikit-learn's [Definition Support Vector Regression](http://scikit-learn.org/stable/modules/svm.html#svm-implementation-details) presents support vector machines (SVMs) as a set of supervised learning methods used for classification, regression and outliers detection. The method of SVM can be extended to solve regression problems. Similar to SVM, the model produced by Support Vector Regression depends only on a subset of the training data, because the cost function for building the model ignores any training data close to the model prediction.

### Support vector clustering (SVC)

With no labels at all, the support vectors can still outline where the data lives. The same notes are in [SVM CLUSTERING](/predictive-ml/clustering-algorithms#svm-clustering).

The support vector clustering [paper](http://www.jmlr.org/papers/volume2/horn01a/horn01a.pdf) is the original method, and the [short explanation](https://www.quora.com/Is-it-possible-to-use-SVMs-for-unsupervised-learning-density-estimation) is the Quora answer on whether SVMs can be used for unsupervised learning and density estimation.

### Regularization and influence

Every variant above has the same two knobs, C and gamma, and they are how an SVM is regularized. The same notes are in [Regularization](/predictive-ml/regularization).

Regularization here is basically punishment for overfitting and raising the non- linear class points higher and lower. [How does regularization look like in SVM](https://datascience.stackexchange.com/questions/4943/intuition-for-the-regularization-parameter-in-svm) - controlling ‘C’ - asks how varying the regularization parameter changes the decision boundary for a non-separable dataset. [The best explanation about Gamma (and C) in SVM!](https://www.quora.com/What-are-C-and-gamma-with-regards-to-a-support-vector-machine) is the same Quora answer on C and gamma as above.

### [Intuition for regularization in SVM](https://datascience.stackexchange.com/questions/4943/intuition-for-the-regularization-parameter-in-svm)

Once the knobs are understood, they still have to be set, and the standard way is a grid. The same notes are in [Regularization](/predictive-ml/regularization).

[Grid search for SVM Hyper parameters](http://docs.opencv.org/3.0-beta/modules/ml/doc/support_vector_machines.html) - in openCV. [Example in log space](https://stackoverflow.com/questions/29128074/choosing-the-best-svm-kernel-type-and-parameters-using-opencv-on-python) is the Stack Overflow question on choosing the kernel type and parameters with OpenCV's cv2.SVM.train\_auto in Python.

The LIBSVM guide gives the grid itself, [I.e., (for example](https://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf), C = 2^-5, 2 ^-3,..., 2^15, γ = 2^-15, 2 ^-13,..., 2^3 ). There are heuristic methods that skip some search options. However, no need for heuristics, computation-time is small, grid can be paralleled and we dont skip parameters. Search complexity is controlled using a two tier grid, coarse grid and then fine tune.

### [Overfitting advice for SVM:](https://stats.stackexchange.com/questions/35276/svm-overfitting-curse-of-dimensionality)

A well-tuned grid can still overfit when the data is small and wide, which is exactly the case in the question behind this heading: 120 samples with 1000-200,000 features, and how SVM handles overfitting, if at all.

scikit-learn's RBF SVM parameters example illustrates the effect of the parameters gamma and C of the Radial Basis Function (RBF) kernel SVM; the regularization parameter C is the [penalty example](http://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html). In non linear kernels, the controls are the kernel choice and the kernel parameters. For [RBF](http://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html) - gamma, low and high values are far and near influence.

The Great [Tutorial at LIBSVM](https://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf) adds the kernel notes. RBF is a reasonable first choice when the relation between class labels and attributes is nonlinear. A special case of C can make this similar to linear kernel (only! After finding C and gamma), and certain parameters makes it behave like the sigmoid kernel. The notes go on: less hyperparameters than RBF kernel, and 0 \<Kij <1 unlike other kernels where the degree is 0\<k\<infinity. Sigmoid is not valid under some parameters. DON'T USE when the #features is very large, use linear.


# Nearest Neighbors

Nearest neighbours classify a point by the points closest to it, which is simple until the data is large and high-dimensional. This page answers how to do that search at scale: first a library for approximate search, then the question of the optimal K, then the benchmarks that compare libraries up to billion-scale search.

### KNN

The method needs a distance, so it depends on how features are scaled, and it is a common interview topic. The same notes are in [Interview questions](/ai-product/data-science-management#interview-questions) and [Normalization & Scaling](/data/processing-intro/normalization-and-scaling).

The first problem is speed. [Nearpy](https://github.com/pixelogik/NearPy) is a Python framework on github for fast (approximated) nearest neighbour search in large, high-dimensional data sets using different locality-sensitive hashes: knn in scale! The second problem is finding the optimal K, how many neighbours should vote. Once several libraries can do the search, they have to be compared: the [Benchmark of nearest neighbours libraries](https://github.com/erikbern/ann-benchmarks/) is erikbern's ann-benchmarks, benchmarks of approximate nearest neighbor libraries in Python. At the far end of scale, [billion scale aprox nearest neighbour search](https://big-ann-benchmarks.com/) is the NeurIPS'23 Competition Track: Big-ANN.


# Regularization

A model that fits noise looks great on training data and fails on anything new; regularization answers that with a penalty added to the loss so such a model is discouraged. The page starts with the definition and lambda, then the vector norms the penalties are built from, then L1 against L2 and why L1 leads to sparsity, then the priors behind both penalties, and ends with how regularization shows up in an SVM.

The same notes are in [DROPOUT LAYERS IN KERAS AND GENERAL](/deep-learning/deep-neural-nets#dropout-layers-in-keras-and-general), [Follow the regularized leader](/decision-intelligence/follow-the-regularized-leader), [Intuition for regularization in SVM](/predictive-ml/linear-separator-algorithms#intuition-for-regularization-in-svm), and [Regularization and influence](/predictive-ml/linear-separator-algorithms#regularization-and-influence).

## What regularization is

The definition starts from the loss a model already minimizes. [Regularization in linear regression](https://datanice.github.io/machine-learning-101-what-is-regularization-interactive.html) is the interactive explanation behind this section: to find the best model, we define a loss or cost function that describes how well the model fits the data, and we try to minimize it. A complex model that also fits the noise is overfitted, so we penalize it by adding a complexity term. That term adds a bigger loss for more complex models.

Lambda sets how strong that penalty is. A bigger lambda rules out high-complexity models (degree 3), because the penalty is stronger. A smaller lambda rules out models with high training error, such as a linear model on non-linear data? (degree 1). The optimum is in between (degree 2).

### Cost functions

Since the penalty is added to a cost function, it helps to know where that cost function comes from. ([I did not watch this](http://web.archive.org/web/20180309202629/https://www.coursera.org/learn/machine-learning/lecture/db3jS/model-representation), but here Andrew Ng talks about cost functions.)

## Vector norms

The complexity term is usually a norm of the weights, so the next step is to recall what a vector norm is, because L1 and L2 are norms. [A rehearsal on vector normalization](http://mathworld.wolfram.com/VectorNorm.html) answers it: for L1, L2, L3, L4, and so on, what is the norm? In some cases it is the absolute value. The figure below draws those norms side by side, copied from the original hosted image.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-c7ba921bace9eff5039cfba0875f01826fee9291%2Fgimg-a495d70a0f5b.png?alt=media" alt=""><figcaption><p>Vector norms for L1, L2, L3, L4, and so on.</p></figcaption></figure>

### ISO surfaces, the Lp norm, and sparseness

The shapes in that figure are ISO surfaces, and their corners are where sparseness will come from later. Watch this. It also explains ISO surfaces, the Lp norm, and sparseness.

{% embed url="<https://www.youtube.com/watch?v=sO4ZirJh9ds>" %}

## L1 and L2

With the norms in hand, the two penalties people actually use are L1 and L2. The same notes are in [FEATURE SELECTION](/data/analytics-intro/dependence-and-selection#feature-selection).

[L1, for sparse models](https://stats.stackexchange.com/questions/45643/why-l1-norm-for-sparse-models) is the question every reader of linear regression books eventually asks: the formulas are known, but why does the L1 norm enforce sparsity in models?

[L1 versus L2, with some formulas](https://towardsdatascience.com/l1-and-l2-regularization-methods-ce25e7fc831c) is an article on understanding L1 and L2 regularization.

### As a loss and as regularization

The same two norms can be the loss itself or the penalty on top of it, and they behave differently in each role. (What is the difference, and what are the features?) [L1 versus L2](http://web.archive.org/web/20260102124606/http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/) as a loss function and as regularization.

{% columns %}
{% column %}
L1 moves the regressor faster. It selects features by driving coefficients to zero. With sparse algorithms it is computationally efficient. With others it is not, so use L2.
{% endcolumn %}

{% column %}
L2 moves more slowly. It does not make the coefficients sparse, and it is computationally efficient.
{% endcolumn %}
{% endcolumns %}

## Why L1 leads to sparsity

The columns claim that L1 drives coefficients to zero; this section shows why. Why does L1 lead to sparsity? There is an [Intuition](https://www.quora.com/Why-is-L1-regularization-supposed-to-lead-to-sparsity-than-L2) for it, and [some of the math](https://www.quora.com/What-is-the-difference-between-L1-and-L2-regularization) is in the Quora question on the difference between L1 and L2 regularization, how it solves overfitting, and which regularizer to use when. The figure shows where the hypothesis meets the L1 and L2 constraints, copied from the original hosted image.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-f21ad67a83cb9c97b5be4f70f48183b5e599739e%2Fgimg-547e6ce30a8a.png?alt=media" alt=""><figcaption><p>Where the hypothesis meets the L1 and L2 constraints.</p></figcaption></figure>

L1 and L2 regularization add constraints to the optimization problem. The curve H0 is the hypothesis. The solution is the set of points where H0 meets the constraints. In L2 the hypothesis is tangential to $$||w||\_2$$. The point of intersection has both an x1 component and an x2 component. In L1, because of the shape of $$||w||\_1$$, the viable solutions are limited to the corners of the axis, for example x1, so x2 = 0. The solution has eliminated the role of x2, which leads to sparsity.

This extends to higher dimensions, which is why L1 regularization leads to solutions in which many of the variables are 0. In other words, L1 regularization leads to sparsity. It is also treated as feature selection. With LibSVM, the recommendation is to select features before using the SVM, and to use L2 instead.

### A one-dimensional intuition

The picture above is geometric; a second intuition, in one dimension, shows why L1 can land exactly on zero. [L1 sparsity, intuition 2](https://www.quora.com/What-is-the-difference-between-L1-and-L2-regularization) comes from the same Quora question. For simplicity, consider the one-dimensional case.

With L2, the regularized loss $$F(x) = f(x) + \lambda ||x||^2$$ is smooth. The optimum is the stationary point, where the derivative is 0. That stationary point of F can become very small as λ increases, but it still will not be 0 unless $$f'(0) = 0$$.

With L1, the regularized loss $$F(x) = f(x) + \lambda ||x||$$ is not smooth. It has a kink, a minimum corner, at 0, and it is not differentiable there. Optimization theory says the optimum of a function is either a point where the derivative is 0 or one of the irregularities, such as a corner or a kink. So the optimal point of F can be 0 even when 0 is not the stationary point of f. If λ is large enough it is 0, which is a stronger regularization effect. Below is a graphical illustration, copied from the original hosted image.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-1da70c01f8995732b05e5bc81d085583eecba68b%2Fgimg-a48abb650e77.png?alt=media" alt=""><figcaption><p>A graphical illustration of the one-dimensional L1 and L2 losses.</p></figcaption></figure>

#### In more than one dimension

The kink argument carries over feature by feature. In more than one dimension: if a feature is not important, the loss it contributes is small, and the non-differentiable regularization effect turns it off.

### Another intuition and formulation

For readers who want the result once more in a different form, the Stack Exchange question on why L1 norm gives sparse models has [An intuition and a formulation, which is pretty good:](https://stats.stackexchange.com/questions/45643/why-l1-norm-for-sparse-models)

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-babd581bdf491642712b462e35be4a00a99b867e%2Fgimg-5f784d0a3b5c.png?alt=media" alt=""><figcaption><p>An intuition and a formulation.</p></figcaption></figure>

## Priors

Penalties can also be read as beliefs about the weights before seeing data: L2 as a Gaussian prior, and L1 as a Laplacean prior. The Stack Exchange question asks how one goes analytically from L2 regularization to a Gaussian prior: L2 regularization is [equivalent to a Gaussian prior](https://stats.stackexchange.com/questions/163388/l2-regularization-is-equivalent-to-gaussian-prior). The figure is that derivation, copied from the original hosted image.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-31089d492a139a3986808a9cdf97386ff717335a%2Fgimg-cb9d4cd1a22d.png?alt=media" alt=""><figcaption><p>L2 regularization as a Gaussian prior.</p></figcaption></figure>

The same question covers the other penalty: L1 regularization is [equivalent to a Laplacean prior](https://stats.stackexchange.com/questions/163388/l2-regularization-is-equivalent-to-gaussian-prior) (same link as above).

> Similarly the relationship between the L1 norm and the Laplace prior can be understood in the same fashion. Take a Laplace prior instead of a Gaussian prior, combine it with your likelihood, and take the logarithm.

## Regularization in an SVM

The last stop is a model where the penalty hides under another name, the parameter C. [How does regularization look in an SVM?](https://datascience.stackexchange.com/questions/4943/intuition-for-the-regularization-parameter-in-svm) asks how varying it changes the decision boundary for a non-separable dataset. It comes down to controlling `C`.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* (Difference between? And features of) L1 vs L2 as loss function and regularization. This address no longer opens: <http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/>
* (did not watch) but here is andrew ng talks about cost functions. This address no longer opens: <https://www.coursera.org/learn/machine-learning/lecture/db3jS/model-representation>


# Probabilistic Models

Instead of drawing a boundary between classes, a probabilistic model says how likely each answer is, and for sequences, how likely each next state is. The page starts with naive Bayes and Bayesian belief networks, moves to Markov models and the hidden Markov model, then to input-output HMMs, and ends with conditional random fields.

### PROBABILISTIC ALGORITHMS

The path runs from the simplest probabilistic classifier to sequence models where each label depends on the one before it.

#### NAIVE BAYES

Naive Bayes is the entry point: apply Bayes' rule and assume the features are independent. A Vidhya piece on NB used to open this list; that address no longer opens and is kept at the end of the page. [Baysian tree](https://github.com/UBS-IB/bayesian_tree) is the UBS-IB bayesian\_tree repository on GitHub. Jake VanderPlas's Python Data Science Handbook chapter, In Depth: Naive Bayes Classification, covers [NB, GNB, multi nominal NB](https://jakevdp.github.io/PythonDataScienceHandbook/05.05-naive-bayes.html).

#### BAYES, BAYESIAN BELIEF NETWORKS

Naive Bayes is one fixed use of Bayes theorem; a belief network lets the dependencies between variables be drawn explicitly. [Mastery on bayes theorem](https://machinelearningmastery.com/bayes-theorem-for-machine-learning/) is MachineLearningMastery's gentle introduction to Bayes theorem for machine learning. [Introduction To BBS](https://codesachin.wordpress.com/2017/03/10/an-introduction-to-bayesian-belief-networks/) is Sachin Joglekar's blog post introducing Bayesian belief networks. A complementing SLIDE presentation that shows how to build the network’s tables, and a very nice presentation regarding BBS, used to follow; both are kept at the end of the page.

Fitting those tables is an estimation problem. [Maximum Likelihood](http://mathworld.wolfram.com/MaximumLikelihood.html) (log likelihood) - proofs for bernoulli, normal, poisson. For [Another example](https://codesachin.wordpress.com/2017/03/10/an-introduction-to-bayesian-belief-networks/), the same Sachin Joglekar post works one through.

#### MARKOV MODELS

Belief networks have no notion of time; Markov models add it, and they need two words kept straight first. Random vs Stochastic ([here](https://math.stackexchange.com/questions/114373/whats-the-difference-between-stochastic-and-random) and [here](https://math.stackexchange.com/questions/569951/what-is-the-difference-between-a-random-vector-and-a-stochastic-process)): the first question asks what the difference between stochastic and random is, and the second asks how a random vector differs from a stochastic process. A variable is 'random'. A process is 'stochastic'.

Apart from this difference the two words are synonyms

In other words, a random vector is a generalization of a single random variables to many, and a stochastic process is a sequence of random variables, or a sequence of random vectors (and then you have a vector-stochastic process).

(What is a Markov Model?) A Markov Model is a stochastic(random) model which models temporal or sequential data, i.e., data that are ordered. It provides a way to model the dependencies of current information (e.g. weather) with previous information. It is composed of states, transition scheme between states, and emission of outputs (discrete or continuous). Several goals can be accomplished by using Markov models: learn statistics of sequential data, do prediction or estimation, and recognize patterns.

The weather version makes it concrete. ([sunny cloudy explanation](http://techeffigytutorials.blogspot.co.il/2015/01/markov-chains-explained.html)) Markov Chains is a probabilistic process, that relies on the current state to predict the next state. To be effective the current state has to be dependent on the previous state in some way: if it looks cloudy outside, the next state we expect is rain. If the rain starts to subside into cloudiness, the next state will most likely be sunny. Not every process has the Markov Property, such as the Lottery, this weeks winning numbers have no dependence to the previous weeks winning numbers.

The tutorial then walks these steps:

1. They show how to build an order 1 markov table of probabilities, predicting the next state given the current.
2. Then it shows the state diagram built from this table.
3. Then how to build a transition matrix from the 3 states, i.e., from the probabilities in the table
4. Then how to calculate the next state using the “current state vector” doing vec\*matrix multiplications.
5. Then it talks about the setting always into the rain prediction, and the solution is using two last states in a bigger table of order 2. He is not really telling us why the probabilities don't change if we add more states, it stays the same as in order 1, just repeating.

#### MARKOV MODELS / HIDDEN MARKOV MODEL

In a plain Markov chain the state is visible; in a hidden Markov model only its effects are. The same notes are in [Timeseries](/predictive-ml/forecasting).

HMM tutorials

The tutorials that used to lead this section, a four-part HMM tutorial (Part 1, 2, 3, 4), a Medium intro to HMM / MM, and HMM with sklearn and networkx, no longer open and are kept at the end of the page. What remains from the Medium thread is the [Paper like example](https://medium.com/@kangeugine/hidden-markov-model-7681c22f5b9), a Medium post on the hidden Markov model.

HMM variants

Implementations come next. [Stack exchange on hmm](https://datascience.stackexchange.com/questions/8460/python-library-to-implement-hidden-markov-models) asks which stable, well-documented Python library can implement Hidden Markov Models. [HMM LEARN](https://github.com/hmmlearn/hmmlearn) is hmmlearn, Hidden Markov Models in Python, with scikit-learn like API (sklearn, still being developed). [Pomegranate](https://pomegranate.readthedocs.io/en/latest/) is the pomegranate documentation, and it covers more than HMMs:

1. General mixture models
2. Hmm
3. Basyes classifiers and naive bayes
4. Markov changes
5. Bayesian networks
6. Markov networks
7. Factor graphs

There is also [GHMM with python wrappers](http://ghmm.org/), and [Hmms](https://github.com/lopatovsky/HMMs) is lopatovsky's continuous-time Hidden Markov Model on GitHub (old).

With the tools listed, the idea itself: HMM ([what is? And why HIDDEN?)](https://youtu.be/jY2E6ExLxaw?t=27m38s) - the idea is that there are things that you CAN OBSERVE and there are things that you CAN'T OBSERVE. From the things you OBSERVE you want to INFER the things you CAN'T OBSERVE (HIDDEN). I.e., you play against someone else in a game, you don't see their choice of action, but you see the result.

In Python, the hmmlearn [code](https://github.com/hmmlearn/hmmlearn) is the main option. HMMs were previously part of [sklearn ](http://scikit-learn.sourceforge.net/stable/modules/hmm.html)Python previously part of the library, until the sklearn.hmm module was deprecated for no longer matching the project's scope and API. For sequence labeling, Python [seqLearn](http://larsmans.github.io/seqlearn/reference.html) is the seqlearn API reference.

This youtube video [part1](https://www.youtube.com/watch?v=TPRoLreU9lA) - explains about the hidden markov model. It shows the visual representation of the model and how we go from that the formula:

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-6e59660f9a9b370b8007ab0e279cf39139d4c4ee%2Fgimg-be8663091250.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

It breaks down the formula to three parts, which the next figure (also copied from the original hosted image) writes out:

* transition probability formula - the probability of going from Zk to Zk+1
* emission probability formula - the probability of going from Zk to Xk
* (Pi) Initial distribution - the probability of Z1=i for i=1..m

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-bb72cdfaed2cc79cd351de37ef4cc95b66a9a3b3%2Fgimg-b1373985e3c7.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

In [part2](https://www.youtube.com/watch?v=M_IIW0VYMEA) of the video, mathematicalmonk's Hidden Markov models (part 2) continues from there.

\* HMM in weka, with github, working on 7.3, not on 9.1; that link no longer opens and is kept at the end of the page.

If the formulas are still opaque, simpler explanations help:

1. Probably the simplest explanation of Markov Models and HMM as a “game” - [link](http://www.fejes.ca/EasyHMM.html)
2. This [video](https://www.youtube.com/watch?v=jY2E6ExLxaw) explains that building blocks of the needed knowledge in HMM, starting probabilities P0, transitions and emissions (state probabilities)

This [post](https://www.quora.com/What-is-a-simple-explanation-of-the-Hidden-Markov-Model-algorithm) This explains HMM and ties our understanding together, as the Quora answer to what a simple explanation of the Hidden Markov Model algorithm is. [A cute explanation on quora](https://www.quora.com/What-is-a-simple-explanation-of-the-Hidden-Markov-Model-algorithm) from that thread comes with these figures:

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-49fa597a6e948e4ef8e20c0cd130e1e8783c550c%2Fgimg-37410fd4f84c.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

This is the iconic image of a Hidden Markov Model. There is some state (x) that changes with time (markov). And you want to estimate or track it. Unfortunately, you cannot directly observe this state (hidden). That's the hidden part. But, you can observe something correlated with the state (y).

OBSERVED DATA -> INFER -> what you CANT OBSERVE (HIDDEN).

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-e1bccdc5e80d44c062d1355f322f8f3080826de4%2Fgimg-876498fdeeba.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

Considering this model, P(X0) is the initial state for happy or sad, P(Xt | X t-1) is the transition model from time-1 to time, and P(Yt | Xt) is the observation model for happy and sad (X) in 4 situations (w, sad, crying, facebook). The last figure fills in those tables.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-9c9efde863e9fe35145fb671b995106398d5d2ce%2Fgimg-4293b4ed9815.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

#### INPUT OUTPUT HMM (IOHMM)

A plain HMM only emits outputs; an input-output HMM also conditions on inputs. The [Incomplete python code](https://github.com/Mogeng/IOHMM) for unsupervised / semi-supervised / supervised IOHMM - training is there, prediction is missing. The reference behind it is Machine learning - a probabilistic approach, david barber; that link no longer opens and is kept at the end of the page.

#### CONDITIONAL RANDOM FIELDS (CRF)

HMMs model how observations are generated; a CRF models the labels given the observations directly, which is why it shows up in templatization and NER. The same notes are in [CRF for templatization](/predictive-ml/templatization#crf-for-templatization), [Named Entity Recognition (NER)](/natural-language-processing-nlp/named-entity-recognition-ner), and [Timeseries](/predictive-ml/forecasting).

[Make sense intro to CRF, comparison against HMM ](https://medium.com/ml2vec/overview-of-conditional-random-fields-68a2a20fa541)is Ravish Chawla's overview: CRFs are a discriminative model for predicting sequences that use contextual information from previous labels. [HMM, CRF, MEMM](https://medium.com/@Alibaba_Cloud/hmm-memm-and-crf-a-comparative-analysis-of-statistical-modeling-methods-49fc32a73586) is Alibaba Cloud's comparative analysis of those three statistical modeling methods. [Another crf article](https://medium.com/@phylypo/nlp-text-segmentation-using-conditional-random-fields-e8ff1d2b6060) is Phylypo Tum on text segmentation, showing how CRF, a special case of the log-linear model, solves the label bias issue of MEMM. The neural network CRF, [NNCRF](https://medium.com/@Akhilesh_k_r/neural-networks-conditional-random-field-crf-973712a0fd30), builds the sequence probability from the softmax outputs the network assigns to each label. The author's list had a fifth item, another one, without an address.

For code, [scikit-learn inspired API for CRFsuite](https://github.com/TeamHG-Memex/sklearn-crfsuite) is TeamHG-Memex/sklearn-crfsuite, [Sklearn wrapper](https://github.com/supercoderhawk/sklearn-crfsuite) is the supercoderhawk fork of the same scikit-learn inspired API, and [Python crfsuite](https://github.com/scrapinghub/python-crfsuite) is scrapinghub's python binding for crfsuite. [Pycrf suite vidahya](https://www.analyticsvidhya.com/blog/2018/08/nlp-guide-conditional-random-fields-text-classification/) is the Analytics Vidhya complete guide to text classification using conditional random fields in Python.

Two last references close the loop back to Markov models. What is a Markov Model?) The Introduction to Markov Models notes show how to establish the transition probabilities between states from collected data: <http://cecas.clemson.edu/~ahoover/ece854/refs/Ramos-Intro-HMM.pdf>. Another one is Freddy Boulton's CRF tutorial in PyTorch, which starts from two identical dice, one fair and one not: <https://towardsdatascience.com/conditional-random-fields-explained-e5b8256da776>

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

\- Vidhya on NB. This address no longer opens: <https://towardsdatascience.com/my-secret-sauce-to-be-in-top-2-of-a-kaggle-competition-57cff0677d3c> - very nice presentation. This address no longer opens: <http://chem-eng.utoronto.ca/\\~datamining/Presentations/Bayesian\\_Belief\\_Network.pdf> - 1. This address no longer opens: <http://gekkoquant.com/2014/05/18/hidden-markov-models-model-description-part-1-of-4/> - 2. This address no longer opens: <http://gekkoquant.com/2014/05/26/hidden-markov-models-forward-viterbi-algorithm-part-2-of-4/> - 3. This address no longer opens: <http://gekkoquant.com/2014/09/07/hidden-markov-models-examples-in-r-part-3-of-4/> - 4. This address no longer opens: <http://gekkoquant.com/2015/02/01/hidden-markov-models-trend-following-sharpe-ratio-3-1-part-4-of-4/> - Intro to HMM. This address no longer opens: <https://towardsdatascience.com/introduction-to-hidden-markov-models-cd2c93e6b781> - HMM with sklearn and networkx. This address no longer opens: <http://www.blackarbs.com/blog/introduction-hidden-markov-models-python-networkx-sklearn/2/9/2017> - HMM in weka, with github, working on 7.3, not on 9.1. This address no longer opens: <http://www.doc.gold.ac.uk/\\~mas02mg/software/hmmweka/index.html> - Machine learning - a probabilistic approach, david barber.. This address no longer opens: <https://pdfs.semanticscholar.org/a632/9a41ee67fae978ccac1e37370f074497a4fe.pdf> - complementing SLIDE presentation. This address no longer opens: <https://www.slideshare.net/GiladBarkan/bayesian-belief-networks-for-dummies>


# Regression

When the target is a continuous number rather than a class, the questions become which regressor to fit and how to judge its error. The page first walks regression libraries and algorithms, from sk-lego and lightning through CART, SVR, neural regression, and logistic regression, then kernel regression, and ends with the error metrics used to judge them, R2, RMSE, and related measures.

The same notes are in [Feature Types](/data/datasets-intro/feature-types), [Normalization & Scaling](/data/processing-intro/normalization-and-scaling), [NYC TAXI](/ai-product/examples), [REGRESSION ALGORITHMS](#regression-algorithms), and [SUPPORT VECTOR REGRESSION (SVR)](/predictive-ml/linear-separator-algorithms#support-vector-regression-svr).

### REGRESSION ALGORITHMS

The algorithms come first, starting with tools that bend a linear regressor to fit data that is not linear. The same notes are in [Regression](/predictive-ml/regression) and [SUPPORT VECTOR REGRESSION (SVR)](/predictive-ml/linear-separator-algorithms#support-vector-regression-svr).

Sk-lego is used to fit with intervals a linear regressor on top of non linear data (its interval encoder page is kept at the end of this page). The figure shows that fit, copied from the original hosted image.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-a1790aa33a870cc571bc9b9a628428b15bd1681f%2Fgimg-06e88d030c4f.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

The same library can also force the fit to be monotonic, Sk-lego monotonic:

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-d9b527d7439b95ab9d71c864adbe27f21a0a5d37%2Fgimg-67779142a4d9.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

When the data is large, [Lightning](https://github.com/scikit-learn-contrib/lightning) - lightning is a library for large-scale linear classification, regression and ranking in Python.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-28378ca6a6f9473393a3d246988f70d6c29308ea%2Fgimg-31ab574359c6.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

Linear regression itself is marked TBC. Beyond it, most classifiers have a regression twin. CART - [classification and regression tree](http://www.simafore.com/blog/bid/62482/2-main-differences-between-classification-and-regression-trees), basically the diff between classification and regression trees - instead of IG we use sum squared error.

The same notes are in [CART TREES](/predictive-ml/decision-trees#cart-trees).

SVR is regression based svm, with kernel only. [NNR](https://deeplearning4j.org/linear-regression)- regression based NN, one output node, points at the Eclipse Deeplearning4j project.

Logistic regression is the odd one out, a regressor used to classify. [LOGREG](http://www.statisticssolutions.com/what-is-logistic-regression/) - Logistic regression - is used as a classification algo to describe data and to explain the relationship between one dependent binary variable and one or more nominal, ordinal, interval or ratio-level independent variables. Output is BINARY. I.e., If the likelihood of killing the bug is > 0.5 it is assumed dead, if it is < 0.5 it is assumed alive. It assumes a binary outcome, assumes no outliers, and assumes no intercorrelations among predictors (inputs?).

Regression Measurements:

Once a regressor is fitted, the first number people check is R^2, and it can mislead. The Minitab blog post on five reasons why your R-squared can be too high lists [several reasons it can be too high.](http://blog.minitab.com/blog/adventures-in-statistics-2/five-reasons-why-your-r-squared-can-be-too-high) The author's short version: too many variables, overfitting, and in time series, seasonality trends can cause this. The other common pair is [RMSE vs MAE](https://medium.com/human-in-a-machine-world/mae-and-rmse-which-metric-is-better-e60ac3bde13d), a post on which of those two most common accuracy metrics for continuous variables is better, noting that publications now mostly use RMSE or some version of R-squared.

#### KERNEL REGRESSION

All of the above fit one global function; kernel regression instead predicts each point from its neighbours. [Gaussian Kernel Regression](http://mccormickml.com/2014/02/26/kernel-regression/) does–it takes a weighted average of the surrounding points.

The key parameter is the variance, sigma^2. Informally, this parameter will control the smoothness of your approximated function. Smaller values of sigma will cause the function to overfit the data points, while larger values will cause it to underfit. There is a proposed method to find sigma in the post! Gaussian Kernel Regression is equivalent to creating an RBF Network with the following properties: - described in the post, and drawn in the figure below, copied from the original hosted image.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-50a39e81bedbc8ecde1017cb1b4e791eaf4ae8c2%2Fgimg-1489877d6d16.png?alt=media" alt=""><figcaption><p>Gaussian kernel regression.</p></figcaption></figure>

## Metrics

Whatever regressor is chosen, it is judged by an error metric, and the first one is the coefficient of determination, [R2](https://en.wikipedia.org/wiki/Coefficient_of_determination). The Medium reading on the rest is a short series: [1](https://medium.com/data-science/regression-an-explanation-of-regression-metrics-and-what-can-go-wrong-a39a9793d914) is Divyanshu Mishra's explanation of regression metrics and what can go wrong, [2](https://medium.com/@george.drakos62/how-to-select-the-right-evaluation-metric-for-machine-learning-models-part-1-regrression-metrics-3606e25beae0) and [3](https://medium.com/@george.drakos62/how-to-select-the-right-evaluation-metric-for-machine-learning-models-part-2-regression-metrics-d4a1a9ba3d74) are parts 1 and 2 of how to select the right evaluation metric for regression, and [4](https://medium.com/usf-msds/choosing-the-right-metric-for-machine-learning-models-part-1-a99d7d7414e4) is Alvira Swalin's part 1 of choosing the right metric for evaluating machine learning models.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Medium 1. This address no longer opens: <https://towardsdatascience.com/regression-an-explanation-of-regression-metrics-and-what-can-go-wrong-a39a9793d914>

3. Tutorial. This address no longer opens: <https://www.dataquest.io/blog/understanding-regression-error-metrics/>

* Sk-lego. This address no longer opens: <https://scikit-lego.readthedocs.io/en/latest/preprocessing.html#Interval-Encoders>


# Decision Trees

A decision tree answers a prediction by asking one feature question at a time, so the real questions are how to choose each split and when to stop. The page starts with split criteria and visualizing forests, then CART with SSE, Gini, early stopping, and pruning, then KD-trees for nearest neighbours, and ends with the forest variants built from trees: random forests and extremely randomized trees.

The same notes are in [Hoeffding tree](/decision-intelligence/incremental-learning#hoeffding-tree), [IMBALANCED DATASETS](/data/datasets-intro/datasets#imbalanced-datasets), [Interview questions](/ai-product/data-science-management#interview-questions), and [Unbalanced labels](/problem-framing/label-algorithms#unbalanced-labels).

The split criterion is the first choice, and it matters most when the classes are imbalanced. Evgeni Dubov's post on classifying imbalanced data starts from the usual sklearn RandomForestClassifier setup and proposes [Using hellinger distance to split supervised datasets, instead of gini and entropy. Claims better results.](https://medium.com/@evgeni.dubov/classifying-imbalanced-data-using-hellinger-distance-f6a4330d6f9a) Once a forest is trained, it helps to look inside it. Visualize decision forests: [forests](https://medium.com/data-science/how-to-visualize-a-decision-tree-from-a-random-forest-in-python-using-scikit-learn-38ad2d75f21c) is Will Koehrsen's guide to visualizing a decision tree from a random forest in Python using scikit-learn.

### [CART TREES](http://machinelearningmastery.com/classification-and-regression-trees-for-machine-learning/)

The classical tree algorithm, known by its more modern name CART, is where the split math lives; the heading link is Jason Brownlee's MachineLearningMastery post on classification and regression trees. The same notes are in [REGRESSION ALGORITHMS](/predictive-ml/regression#regression-algorithms) and [Tutorials](/data/analytics-intro/information-theory#tutorials).

The post explains about the similarities and how to measure. which is the best split? based on SSE and GINI (good info about gini here). For classification the Gini cost function is used which provides an indication of how “pure” the leaf nodes are (how mixed the training data assigned to each node is).

Gini = sum(pk \* (1 – pk))

A tree grown until every leaf is pure memorizes the data, so two controls follow. Early stop — 1 sample per node is overfitting, 5-10 are good. Pruning — evaluate what happens if the lead nodes are removed, if there is a big drop, we need it.

### KDTREE

The same split-by-feature idea also organizes space for search rather than prediction. Victor Lavrenko's KD tree algorithm: how it works explains it: [Similar to a binary search tree, just by using the median and selecting a feature randomly for each level.](https://www.youtube.com/watch?v=TLxWtXEbtFE) His kNN.15 K-d tree algorithm lecture shows what it is for: [Used to find nearest neighbours.](https://www.youtube.com/watch?v=Y4ZgLlDfKDg) The Quora question on what a kd-tree is used for lists [Many applications of using KD tree, reduce color space, Database key search, etc](https://www.quora.com/What-is-a-kd-tree-and-what-is-it-used-for).

### RANDOM FOREST

Back to prediction: one tree overfits, so a forest averages many, and the forest can also be used as a feature transform. The same notes are in [Ensembles](/predictive-ml/ensembles).

scikit-learn's feature transformations with ensembles of trees fits an ensemble first, then trains a linear model on the resulting features: [Using an ensemble of trees to create a high dimensional and sparse representation of the data and classifying using a linear classifier](http://scikit-learn.org/stable/auto_examples/ensemble/plot_feature_transformation.html#sphx-glr-auto-examples-ensemble-plot-feature-transformation-py)

Imbalance comes back here too. [How do deal with imbalanced data in Random-forest](http://statistics.berkeley.edu/sites/default/files/tech-reports/666.pdf) — the Berkeley paper proposes two ways and shows both improve prediction accuracy:

1. One is based on cost sensitive learning.
2. Other is based on a sampling technique

### EXTRA TREES

Random forests still search for the best split; extremely randomized trees go one step further and randomize the split itself. The same notes are in [Ensembles](/predictive-ml/ensembles).

A comparison between random forest and extra trees used to be linked here; the original article address is kept at the end of the page. Its figure caption reads:

Fig. 1: Comparison of random forests and extra trees in presence of irrelevant predictors. In blue are presented the results from the random forest and red for the extra trees. The results are quite striking: Extra Trees perform consistently better when there are a few relevant predictors and many noisy ones

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-40d29af3d66d82e737d3d616c7f7e312530ffd2f%2Fgimg-a7e3fe15e11e.png?alt=media" alt=""><figcaption><p>Comparison of random forests and extra trees in presence of irrelevant predictors.</p></figcaption></figure>

To see why, the [Difference between RF and ET](https://stats.stackexchange.com/questions/175523/difference-between-random-forest-and-extremely-randomized-trees) thread states that random forest splits are deterministic, while in extremely randomized trees the next split is the best among random uniform splits, and asks what impact that has. [Differences #2](https://stackoverflow.com/questions/22409855/randomforestclassifier-vs-extratreesclassifier-in-scikit-learn) is the same question for RandomForestClassifier vs ExtraTreesClassifier in scikit-learn, read against the Geurts, Ernst, and Wehenkel paper "Extremely randomized trees".

To close, Will Koehrsen's post, a helpful utility for understanding your model, is the place to visualize decision forests: <https://towardsdatascience.com/how-to-visualize-a-decision-tree-from-a-random-forest-in-python-using-scikit-learn-38ad2d75f21c>

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Visualize decision trees. This address no longer opens: <https://towardsdatascience.com/interactive-visualization-of-decision-trees-with-jupyter-widgets-ca15dd312084>
* A comparison between random forest and extra trees. This address no longer opens: <https://www.thekerneltrip.com/statistics/random-forest-vs-extra-tree/>


# Ensembles

One model has one set of mistakes; several models combined can cancel each other's. This page answers how to combine them, starting with an overview of voting, bagging, boosting, and stacking, then the WEKA view of each, then bagging and boosting in detail, and finally gradient boosting, XGBoost, and CatBoost.

The same notes are in [Active Learning](/problem-framing/active-learning), [EXTRA TREES](/predictive-ml/decision-trees#extra-trees), [Interview questions](/ai-product/data-science-management#interview-questions), and [RANDOM FOREST](/predictive-ml/decision-trees#random-forest).

The overview comes first. Toptal's Ensemble Methods: The Kaggle Machine Learning Champion is the (good) [review on voting, bagging, boosting stacking, cascading methodologies](https://www.toptal.com/machine-learning/ensemble-methods-kaggle-machine-learn), built on the point that ensembles often win Kaggle contests. The simplest ensemble is a vote, and sentdex's Combining Algos with a Vote, part 16 of Natural Language Processing With Python and NLTK, shows [How to combine several sklearn algorithms into a voting ensemble](https://www.youtube.com/watch?v=vlTQLb_a564\&list=PLQVvvaa0QuDf2JswnfiGkliBInZnIC4HL\&index=16). One step up from voting is stacking, where a model learns how to combine the others; Sebastian Raschka's mlxtend StackingClassifier is the [Stacking api, MLXTEND](http://rasbt.github.io/mlxtend/user_guide/classifier/StackingClassifier/).

Machine learning Mastery has a run of posts on the same ideas, each with its own outline. Stacking Ensemble for Deep Learning Neural Networks in Python is [stacking neural nets — really good](https://machinelearningmastery.com/stacking-ensemble-for-deep-learning-neural-networks/):

1. Stacked Generalization Ensemble
2. Multi-Class Classification Problem
3. Multilayer Perceptron Model
4. Train and Save Sub-Models
5. Separate Stacking Model
6. Integrated Stacking Model

The ways to merge outputs are in [How to Combine Predictions for Ensemble Learning](https://machinelearningmastery.com/combine-predictions-for-ensemble-learning/):

1. Plurality Voting.
2. Majority Voting.
3. Unanimous Voting.
4. Weighted Voting.

The family of stacking methods is in [Essence of Stacking Ensembles for Machine Learning](https://machinelearningmastery.com/essence-of-stacking-ensembles-for-machine-learning/):

1. Voting Ensembles
2. Weighted Average
3. Blending Ensemble
4. Super Learner Ensemble

Instead of one fixed combination, the choice of models can depend on the input. [Dynamic Ensemble Selection (DES) for Classification in Python](https://machinelearningmastery.com/dynamic-ensemble-selection-in-python/) — Dynamic Ensemble Selection algorithms operate much like DCS algorithms, except predictions are made using votes from multiple classifier models instead of a single best model. In effect, each region of the input feature space is owned by a subset of models that perform best in that region. The post's outline:

1. k-Nearest Neighbor Oracle (KNORA) With Scikit-Learn
2. KNORA-Eliminate (KNORA-E)
3. KNORA-Union (KNORA-U)
4. Hyperparameter Tuning for KNORA
5. Explore k in k-Nearest Neighbor
6. Explore Algorithms for Classifier Pool

A gating model that routes inputs to experts is the same idea made trainable, in [A Gentle Introduction to Mixture of Experts Ensembles](https://machinelearningmastery.com/mixture-of-experts/):

1. Mixture of Experts
2. Subtasks
3. Expert Models
4. Gating Model
5. Pooling Method
6. Relationship With Other Techniques
7. Mixture of Experts and Decision Trees
8. Mixture of Experts and Stacking

The last post in the run explains why weak models are worth combining at all. [Strong Learners vs. Weak Learners in Ensemble Learning](https://machinelearningmastery.com/strong-learners-vs-weak-learners-for-ensemble-learning/) — Weak learners are models that perform slightly better than random guessing. Strong learners are models that have arbitrarily good accuracy.

Weak and strong learners are tools from computational learning theory and provide the basis for the development of the boosting class of ensemble methods.

Most practical ensembles are built from trees. Himanshi Singh's tree-based algorithms tutorial covers decision trees, random forest, ensemble methods, and their implementation in R & python: [Vidhya on trees, bagging boosting, gbm, xgb](https://www.analyticsvidhya.com/blog/2016/04/complete-tutorial-tree-based-modeling-scratch-in-python/#three). For speed, [Parallel grad boost treest](http://zhanpengfang.github.io/418home.html) is the Parallel Gradient Boosting Decision Trees project. Aishwarya Singh's guide to ensemble learning techniques, from simple methods to bagging and boosting and the key algorithms, is [A comprehensive guide to ensembles read!](https://www.analyticsvidhya.com/blog/2018/06/comprehensive-guide-for-ensemble-models/), with this outline:

1. Basic Ensemble Techniques
2. 2.1 Max Voting
3. 2.2 Averaging
4. 2.3 Weighted Average
5. Advanced Ensemble Techniques
6. 3.1 Stacking
7. 3.2 Blending
8. 3.3 Bagging
9. 3.4 Boosting
10. Algorithms based on Bagging and Boosting
11. 4.1 Bagging meta-estimator
12. 4.2 Random Forest
13. 4.3 AdaBoost
14. 4.4 GBM
15. 4.5 XGB
16. 4.6 Light GBM
17. 4.7 CatBoost

For competitions, the [Kaggler guide to stacking](http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/) is stacking in practice, and [Blending vs stacking](https://www.quora.com/What-are-examples-of-blending-and-stacking-in-Machine-Learning) separates the two terms. A Kaggle ensemble guide used to sit here too; its address is kept at the end of the page.

### [Ensembles in WEKA](http://machinelearningmastery.com/use-ensemble-machine-learning-algorithms-weka/)

With the landscape laid out, the WEKA post by Jason Brownlee puts the five main ensembles side by side, since WEKA makes so many of them available. Bagging is random sample selection and multi classifier training. Random forest is random feature selection for each tree, multi tree training. Boosting means creating stumps, each new stump tries to fix the previous error, at last combining results using new data, each model is assigned a skill weight and accounted for in the end. Voting is a majority vote, any set of algorithms within weka, results combined via mean or some other way. Stacking is the same as voting but combining predictions using a meta model is used.

### BAGGING - bootstrap aggregating

Bagging is the first of those to unpack, because it is the simplest. [Bagging](https://www.youtube.com/watch?v=2Mg8QD0F1dQ\&list=PLAwxTw4SYaPnIRwl6rad_mYwEk4Gmj7Mx\&index=192) — best example so far, create m bags, put n’\<n samples (60% of n) in each bag — with replacement which means that the same sample can be selected twice or more, query from the test (x) each of the m models, calculate mean, this is the classification. The figure below draws those bags, copied from the original hosted image.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-cffe2d7c9d0c2a12018914eb1bb705e1007a2283%2Fgimg-f2649232c408.png?alt=media" alt=""><figcaption><p>Bagging.</p></figcaption></figure>

Overfitting — not an issue with bagging, as the mean of the models actually averages or smoothes the “curves”. Even if all of them are overfitted. The next figure shows that smoothing.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-94245a604cb7962da2f2ecf3cd50c7e445af7f75%2Fgimg-2eb3a4109a08.png?alt=media" alt=""><figcaption><p>Bagging and overfitting.</p></figcaption></figure>

A random forest is bagging over trees, and in xgboost it is one round of many parallel trees:

\#Random Forest™ - 1000 trees

bst <- xgboost(data = train$data, label = train$label, max\_depth = 4, num\_parallel\_tree = 1000, subsample = 0.5, colsample\_bytree =0.5, nrounds = 1, objective = "binary:logistic")

### BOOSTING

Bagging trains its models independently; boosting trains them in sequence, each one focused on what the previous ones got wrong. Mastery on using [all the boosting algorithms](https://machinelearningmastery.com/gradient-boosting-with-scikit-learn-xgboost-lightgbm-and-catboost/): Gradient Boosting with Scikit-Learn, XGBoost, LightGBM, and CatBoost

Adaboost: similar to bagging, create a system that chooses from samples that were modelled poorly before.

1. create bag\_1 with n’ features \<n with replacement, create the model\_1, test on ALL train.
2. Create bag\_2 with n’ features with replacement, but add a bias for selecting from the samples that were wrongly classified by the model\_1. Create a model\_2. Average results from model\_1 and model\_2. I.e., who was classified correctly or not.
3. Create bag\_3 with n’ features with replacement, but add a bias for selecting from the samples that were wrongly classified by the model\_1+2. Create a model\_3. Average results from model\_1, 2 & 3 I.e., who was classified correctly or not. Iterate onward.
4. Create bag\_m with n’ features with replacement, but add a bias for selecting from the samples that were wrongly classified by the previous steps.

The figure below shows those reweighted rounds, copied from the original hosted image.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-675051f4fe5d1bc6c9f07f8c8051bc30e6632384%2Fgimg-4ad806b940e4.png?alt=media" alt=""><figcaption><p>Boosting.</p></figcaption></figure>

In xgboost, boosting is the same call with sequential rounds instead of parallel trees:

\#Boosting - 3 rounds

bst <- xgboost(data = train$data, label = train$label, max\_depth = 4, nrounds = 3, objective = "binary:logistic")

The two settings side by side:

RF1000: - max\_depth = 4, num\_parallel\_tree = 1000, subsample = 0.5, colsample\_bytree =0.5, nrounds = 1, nthread = 2

XG: nrounds = 10, max\_depth = 4, eta = 0.5, nthread = 2

### Gradient Boosting Classifier

Gradient boosting generalizes that reweighting into fitting the gradient of a loss, and the practical question is how scikit-learn's GBC compares with XGB. [Loss functions and GBC vs XGB](https://stats.stackexchange.com/questions/202858/loss-function-approximation-with-taylor-expansion) is the question on how XGBoost approximates its loss function with a Taylor expansion, one of the key steps for fast calculation. [Why is XGB faster than SK GBC](https://datascience.stackexchange.com/questions/10943/why-is-xgboost-so-much-faster-than-sklearn-gradientboostingclassifier) starts from XGBClassifier handling 500 trees in 43 seconds while GradientBoostingClassifier handles only 10 trees in about a minute. A good XGB vs GBC tutorial used to be linked here and is kept at the end of the page. [XGB vs GBC](https://stats.stackexchange.com/questions/282459/xgboost-vs-python-sklearn-gradient-boosted-trees) asks whether XGBoost works the same way as sklearn's gradient boosted trees, but faster, or whether there are fundamental differences.

### XGBOOST

That comparison makes XGBoost the next stop. What is XGBOOST? — XGBoost is an optimized distributed gradient boosting system designed to be highly efficient, flexible and portable [#2nd link](http://dmlc.cs.washington.edu/xgboost.html). The first link on that question no longer opens and is kept at the end of the page.

Because boosting keeps fitting the errors, it is fair to ask whether it overfits: [Does it cause overfitting?](https://stats.stackexchange.com/questions/20714/does-ensembling-boosting-cause-overfitting) is a question about accuracy changing when boosting is turned on with 400 ensembles. The [Authors Youtube lecture.](https://www.youtube.com/watch?v=Vly8xGnNiWs) is XGBoost A Scalable Tree Boosting System, June 02, 2016, from Real Data Science USA (formerly DataScience.LA). The code is on [GIT here](https://github.com/dmlc/xgboost): dmlc/xgboost, a scalable, portable and distributed gradient boosting (GBDT, GBRT or GBM) library for Python, R, Java, Scala, C and more. A How to use XGB tutorial on medium (comparison to GBC) is kept at the end of the page.

For hands-on use, the [How to code tutorial](https://www.youtube.com/watch?v=87xRqEAx6CY), short and makes sense, with info about the parameters: threads, rounds, tree height, loss function, error, and cross fold. The [Beautiful Video Class about XGBOOST](https://www.youtube.com/playlist?list=PLZnYQQzkMilqTC12LmnN4WpQexB9raKQG) — mostly practical in jupyter but with some insight about the theory. [Machine learning mastery](http://machinelearningmastery.com/gentle-introduction-xgboost-applied-machine-learning/) has A Gentle Introduction to XGBoost for Applied Machine Learning.

XGBoost can also run from WEKA. WekaMOOC's Advanced Data Mining with Weka lesson on setting up R covers [R Installation in Weka](https://www.youtube.com/watch?v=EGwHXC3baWU\&list=PLm4W7_iX_v4Msh-7lDOpSFWHRYU_6H5Kx\&index=15), then XGBOOST in weka through R

Parameters for weka mlr class.xgboost. The forum thread on them is kept at the end of the page; the reference is the R package manual for xgboost, Extreme Gradient Boosting, the R interface to Chen & Guestrin's implementation:

* <https://cran.r-project.org/web/packages/xgboost/xgboost.pdf>
* Here is an example configuration for multi-class classification:
* weka.classifiers.mlr.MLRClassifier -learner “nrounds = 10, max\_depth = 2, eta = 0.5, nthread = 2”
* classif.xgboost -params "nrounds = 1000, max\_depth = 4, eta = 0.05, nthread = 5, objective = \\"multi:softprob\\"

Copy: nrounds = 10, max\_depth = 2, eta = 0.5, nthread = 2

Special case of random forest using XGBOOST: the vignette for it is kept at the end of the page, and the RF1000 setting above is the same idea.

## CatBoost

XGBoost is not the last word in gradient boosting. CatBoost is based on gradient boosting, a technique developed by Yandex that outperforms many existing boosting algorithms like XGBoost and Light GBM; that is the (great) [what is so special?](https://hanishrohit.medium.com/whats-so-special-about-catboost-335d64d754ae) post. [the fastest algo](https://medium.com/almabetter/catboost-the-fastest-algorithm-c21d44f8b990) is a deep dive into what it actually is, and [a new game in ML](https://affine.medium.com/catboost-a-new-game-of-machine-learning-72a7dcea0ac4) places it among gradient boosted decision trees and random forest, one of the best ML models for tabular heterogeneous datasets. A fourth post, use it here is why, is kept at the end of the page.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Kaggle ensemble guide. This address no longer opens: <https://mlwave.com/kaggle-ensembling-guide/>
* What is XGBOOST?. This address no longer opens: <http://homes.cs.washington.edu/\\~tqchen/2016/03/10/story-and-lessons-behind-the-evolution-of-xgboost.html>
* How to use XGB tutorial on medium (comparison to GBC). This address no longer opens: <https://towardsdatascience.com/boosting-algorithm-xgboost-4d9ec0207d>
* Parameters for weka mlr class.xgboost. This address no longer opens: <http://weka.8497.n7.nabble.com/XGBoost-in-Weka-through-R-or-Python-td40282.html>
* Special case of random forest using XGBOOST. This address no longer opens: <https://github.com/dmlc/xgboost/blob/master/R-package/vignettes/discoverYourData.Rmd#special-note-what-about-random-forests>
* Good XGB vs GBC tutorial. This address no longer opens: <https://towardsdatascience.com/boosting-algorithm-xgboost-4d9ec0207d>
* use it here is why. This address no longer opens: <https://towardsdatascience.com/you-should-use-catboost-heres-why-72f124dcdad7>


# Genetic Algorithms & Genetic Programming

Genetic algorithms and genetic programming are easy to confuse, because both evolve an answer instead of fitting one. The short version is that GA evolves parameters and GP evolves programs or expressions. The page first shows what the two share, in peterjwest's words, and then what separates them, in johnIdol's.

The evolutionary idea also shows up elsewhere in the book. The same notes are in [Learning Classifier Systems](/predictive-ml/learning-classifier-systems), [ML Systems](/deep-learning/meta-learning#ml-systems), and [NEURO EVOLUTION (GA/GP based)](/deep-learning/other-architectures#neuro-evolution-gagp-based).

The question itself comes from the Stack Overflow thread [What is the difference?](https://stackoverflow.com/questions/3819977/what-are-the-differences-between-genetic-algorithms-and-genetic-programming), and the first answer starts with what the two have in common. Genetic programming and genetic algorithms are very similar. They are both used to evolve the answer to a problem, by comparing the fitness of each candidate in a population of potential candidates over many generations. Each generation, new candidates are found by randomly changing (mutation) or swapping parts (crossover) of other candidates. The least 'fit' candidates are removed from the population. That answer is by peterjwest, and the figure below draws the same loop.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-cf9c07be323d2ada6f026eba07dd2d1ea52f567f%2Fgimg-437170434e60.png?alt=media" alt=""><figcaption><p>Genetic algorithms and genetic programming.</p></figcaption></figure>

Once the shared loop is clear, the difference is what a single individual is. Genetic algorithms (GA) are search algorithms that mimic the process of natural evolution, where each individual is a candidate solution: individuals are generally "raw data" (in whatever encoding format has been defined). Genetic programming (GP) is considered a special case of GA, where each individual is a computer program (not just "raw data"). GP explore the algorithmic search space and evolve computer programs to perform a defined task. That distinction is johnIdol's answer in the same thread, and the second figure shows GP sitting inside GA.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-49f0e29f4361d6e3203b3c77ee8f7f8cc19bcdac%2Fgimg-b6658c7c8951.png?alt=media" alt=""><figcaption><p>Genetic programming as a special case of GA.</p></figcaption></figure>


# Learning Classifier Systems

Learning Classifier Systems (LCS) sit next to genetic algorithms for the case where the learner is a rule population rather than a single model. The page first defines LCS, then narrows to XCS, the variant with a scikit-learn implementation and a tutorial.

The same notes are in [Genetic Algorithms & Genetic Programming](/predictive-ml/genetic-algorithms-and-genetic-programming).

### LCS

The starting point is the Wikipedia definition, because it names the two parts every LCS has. (WIKI) [Learning classifier systems](https://en.wikipedia.org/wiki/Learning_classifier_system), or LCS, are a paradigm of [rule-based machine learning](https://en.wikipedia.org/wiki/Rule-based_machine_learning) methods that combine a discovery component (e.g. typically a [genetic algorithm](https://en.wikipedia.org/wiki/Genetic_algorithm)) with a learning component (performing either [supervised learning](https://en.wikipedia.org/wiki/Supervised_learning), [reinforcement learning](https://en.wikipedia.org/wiki/Reinforcement_learning), or [unsupervised learning](https://en.wikipedia.org/wiki/Unsupervised_learning)). The learning component is the choice that matters: in supervised learning the training data is labeled with the expected answers, while in unsupervised learning the model finds patterns or structures in unlabeled data.

### XCS

With the general definition in place, XCS is the concrete variant that picks reinforcement learning as its learning component. [XCS](http://hosford42.github.io/xcs/) is a type of [Learning Classifier System (LCS)](http://en.wikipedia.org/wiki/Learning_classifier_system), a [machine learning](http://en.wikipedia.org/wiki/Machine_learning) algorithm that utilizes a [genetic algorithm](http://en.wikipedia.org/wiki/Genetic_algorithm) acting on a rule-based system, to solve a [reinforcement learning](http://en.wikipedia.org/wiki/Reinforcement_learning) problem.

To run it, [Scikit-xcs](https://github.com/UrbsLab/scikit-XCS) is the scikit-learn compatible implementation of XCS, described by its authors as the most popular and best studied learning classifier system algorithm to date. The figure below is from that repository.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-a1655dc32351e2b52b76a98c73ea87dd71676c97%2Fgimg-d7eaf61118ae.png?alt=media" alt=""><figcaption><p>Scikit-xcs.</p></figcaption></figure>

For a guided walk through the same algorithm, the XCS [Tutorial](https://pythonhosted.org/xcs/) goes with the hosford42 XCS package linked above.


# Clustering Algorithms

Clustering is for grouping unlabeled points, and the right method depends on whether the groups are defined by distance, by density, or by must-link / cannot-link constraints. The page starts with the general map and how to pick the number of clusters, moves through the k-means family and GMM, then the density-based methods (DBSCAN, HDBSCAN, OPTICS), and ends with SVM clustering and constrained clustering.

The same notes are in [CLUSTERING METRICS](/predictive-ml/anomaly-detection#clustering-metrics), [CLUSTERING TS](/predictive-ml/time-series-search#clustering-ts), and [UNSUPERVISED](/evals/evaluation-metrics#unsupervised).

The map comes first. [Vidhya on clustering and methods](https://www.analyticsvidhya.com/blog/2016/11/an-introduction-to-clustering-and-different-methods-of-clustering/?utm_source=facebook.com) walks through the different types of clustering techniques in machine learning and how they are used to identify structure in data.

Nearest neighbours are the simplest way to think about "close points belong together", even though KNN itself is a classifier. [KNN](https://www.youtube.com/watch?v=4ObVzTuFivY) is mathematicalmonk's lecture on the k-nearest neighbor classification algorithm, and [intuition 2](https://www.youtube.com/watch?v=UqYde-LULfs) is Thales Sehn Körting's video on how the kNN algorithm works; a thorough explanation 3 used to sit beside them and is kept at the end of the page. The idea is to classify a new sample by looking at the majority vote of its K-nearest neighbours, with k=1 as the special case. Even amount of classes needs an odd K that is not a multiple of the amount of classes in order to break ties.

Every method below still has to answer how many clusters there are. [Determinging the number of clusters, a comparison of several methods, elbow, silhouette etc](https://www.datanovia.com/en/lessons/determining-the-optimal-number-of-clusters-3-must-know-methods/) compares the elbow, silhouette, and gap statistic methods in R with fviz\_nbclust(), plus NbClust’s 26-index majority vote. A good visual example of kmeans / gmm used to be linked here; that address no longer opens and is kept at the end of the page.

Once the cluster count is settled, time series add their own twist: Kmeans with DTW, probably fixed length vectors, using tslearn, is the approach in <https://towardsdatascience.com/how-to-apply-k-means-clustering-to-time-series-data-28d04a8f7da3>. The same notes are in [Dynamic Time Warping (DTW)](/predictive-ml/time-series-search#dynamic-time-warping-dtw).

When the observations do not even have the same length, [Kmeans for variable length](https://medium.com/@iliazaitsev/how-to-classify-a-dataset-with-observations-of-various-length-96fab8e95baf) shows how to use k-means to classify a dataset with a non-fixed number of features. Its example is the Wrist-Worn Accelerometer Dataset: files of 3-dimensional accelerometer measurements, 839 observations in 14 classes, each class a single action by the wearer such as Walking, Drinking, or Eating. The code is in the [notebook](https://github.com/devforfu/Blog/blob/master/trees/scikit_learn.py), part of the i-zaitsev/Blog repository of projects supporting the blog posts.

TOOLS

For running many of these algorithms from one library, the tool is [pyClustering](https://pyclustering.github.io/docs/0.10.1/html/index.html).

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-347c24eec478112b57707fd08939a861fd16fad5%2Fgimg-aef18cd451cd.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

###

### Block Modeling - Distance Matrices

Sometimes the input is not a table of points but a matrix of how every item relates to every other item. The same notes are in [Graph Theory](/predictive-ml/graph-theory).

[Biclustering and spectral co clustering](https://scikit-learn.org/stable/modules/biclustering.html) is the scikit-learn guide to algorithms that cluster the rows and columns of a data matrix at the same time; each resulting bicluster determines a submatrix of the original data. When the matrix is square, [Clustering correlation, or distance matrices.](https://stats.stackexchange.com/questions/138325/clustering-a-correlation-matrix) is Abhishek093's question: given an N\*N correlation matrix of categorical items, how to cluster the N items into M bins so that the items in each bin behave the same.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-f6d2b34b2f78938df7f79e2f500d0e214cd5b846%2Fgimg-14594f9aa550.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

In practice, any of the “precomputed” algorithms in sklearn will take such a matrix, just remember to [do 1-distanceMatrix](https://github.com/scikit-learn/scikit-learn/issues/6787). I.e., using dbscan/hdbscan/optics, you need a dissimilarity matrix.

### [Kmeans](https://github.com/jakevdp/sklearn_pycon2015/blob/master/notebooks/04.2-Clustering-KMeans.ipynb)

From matrices back to points, k-means is the default, and the heading links the PyCon 2015 notebook for it. Its main weakness is that it is sensitive to outliers, which can skew results (because we rely on the mean).

### KMEANS++ / Kernel Kmeans

The fixes to plain k-means start with smarter initialization and kernels. [A comparison of kmeans++ vs kernel kmeans](https://sandipanweb.wordpress.com/2016/08/29/kernel-k-means-and-cluster-evaluation/) is sandipanweb's post on kernel k-means, k-means++, and cluster evaluation. Kernel Kmeans is part of TSLearn.

Choosing k for these comes back to the elbow method. [elbow and mean silhouette](https://www.datanovia.com/en/lessons/determining-the-optimal-number-of-clusters-3-must-know-methods/#elbow-method) jumps to the elbow part of the same Datanovia comparison of elbow, silhouette, gap statistic, and NbClust’s 26-index majority vote. Another version is the elbow on medium using mean distance per cluster from the center. To find the elbow automatically, [Kneed a library to find the knee in a curve](https://github.com/arvkevi/kneed) does knee point detection in Python, and [how to?](https://stackoverflow.com/questions/47623915/how-to-detect-in-real-time-a-knee-elbow-maximal-curvature-in-a-curve) is Marco's question on how to detect a "knee/elbow" (maximal curvature) in a curve in real time.

### [K-mediods](https://en.wikipedia.org/wiki/K-medoids)

If outliers are the problem with the mean, k-medoids replaces the mean with a real point. It is basically k-means with a most center object rather than a center virtual point that was based on mean distance from all points, and we keep choosing medoids samples based on minimised SSE.

k-medoid is a classical partitioning technique of clustering that clusters the data set of n objects into k clusters known a priori. It is more robust to noise and outliers as compared to [k-means](https://en.wikipedia.org/wiki/K-means) because it minimizes a sum of pairwise dissimilarities instead of a sum of squared Euclidean distances. A [medoid](https://en.wikipedia.org/wiki/Medoid) can be defined as the object of a cluster whose average dissimilarity to all the objects in the cluster is minimal, i.e. it is a most centrally located point in the cluster. The cost is that it does Not scale to many samples, its O(K\*n-K)^2, though randomized resampling can assure efficiency and quality.

The video walkthrough is [From youtube (okay video)](https://www.youtube.com/watch?v=OWpRBCrx5-M), and the figure below goes with it.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-b394512f92405d36c749bf59b9bc9a9ca0c938e6%2Fgimg-2e83f10446a7.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

### K-modes

Means and medoids both assume numeric distances, so categorical data needs modes instead. The [git](https://github.com/nicodv/kmodes#huang97) repository describes itself as "Python implementations of the k-modes and k-prototypes clustering algorithms, for clustering categorical data". When the table mixes numeric and categorical columns, [a guide to clustering mixed types](https://bpostance.github.io/posts/clustering-mixed-data/) is Ben Postance's guide to clustering large datasets with mixed data-types \[updated], from his risk, data science and machine learning blog.

### X-means

The k-means variants so far still need k handed to them. X-means estimates it, using BIC to decide the cluster count.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-f27e704e6e6654170b8472e0241c36d507720211%2Fgimg-16c9a657d6e0.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

X-means ([paper](https://www.cs.cmu.edu/~dpelleg/download/xmeans.pdf)) is the method in the figure above. The [Theory](https://stats.stackexchange.com/questions/13103/x-mean-algorithm-bic-calculation-question) behind bic calculation with a formula is Budric's question about the BIC formulas in Dan Pelleg and Andrew Moore's X-means: Extending K-means with Efficient Estimation of the Number of Clusters, starting from the variance equation. For Code, [Calculate bic in k-means](https://stats.stackexchange.com/questions/90769/using-bic-to-estimate-the-number-of-k-in-kmeans?rq=1) is Kam Sen's attempt to compute the BIC for the iris toy data set and reproduce the paper's results.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-c6062e82d36c6260488ad4ee5fdf0e4619cfed05%2Fgimg-b237b0258625.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

### G-means

G-means grows k with a statistical test instead of BIC. G-means [Improves on X-means](https://papers.nips.cc/paper/2526-learning-the-k-in-k-means.pdf) in the paper: The G-means algorithm starts with a small number of k-means centers, and grows the number of centers. Each iteration of the algorithm splits into two those centers whose data appear not to come from a Gaussian distribution using the Anderson Darling test. Between each round of splitting, we run k-means on the entire dataset and all the centers to refine the current solution. We can initialize with just k = 1, or we can choose some larger value of k if we have some prior knowledge about the range of k. G-means repeatedly makes decisions based on a statistical test for the data assigned to each enter. If the data currently assigned to a k-means center appear to be Gaussian, then we want to represent that data with only one center.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-22acbe777019e7f54627565397c768ac58c1aefc%2Fgimg-0cb2440dad14.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

### GMM - Gaussian Mixture Models

G-means already treats clusters as Gaussians; GMM makes that the whole model and lets a point belong to several clusters. [What is GMM](https://datascience.stackexchange.com/questions/14435/how-to-get-the-probability-of-belonging-to-clusters-for-k-means)? In short, it is knn with mean/variance centroids, and a sample can be in several centroids with a certain probability. The answer in that thread lays the two procedures side by side:

Let us briefly talk about a probabilistic generalisation of k-means: the [Gaussian Mixture Model](https://en.wikipedia.org/wiki/Mixture_model)(GMM).

In k-means, you carry out the following procedure:

\- specify k centroids, initialising their coordinates randomly

\- calculate the distance of each data point to each centroid

\- assign each data point to its nearest centroid

\- update the coordinates of the centroid to the mean of all points assigned to it

\- iterate until convergence.

In a GMM, you carry out the following procedure:

\- specify k multivariate Gaussians (termed components), initialising their mean and variance randomly

\- calculate the probability of each data point being produced by each component (sometimes termed the responsibility each component takes for the data point)

\- assign each data point to the component it belongs to with the highest probability

\- update the mean and variance of the component to the mean and variance of all data points assigned to it

\- iterate until convergence

You may notice the similarity between these two procedures. In fact, k-means is a GMM with fixed-variance components. Under a GMM, the probabilities (I think) you're looking for are the responsibilities each component takes for each data point.

The scikit-learn examples then show the model in code. [Gmm code on sklearn](https://scikit-learn.org/stable/auto_examples/mixture/plot_gmm.html#sphx-glr-auto-examples-mixture-plot-gmm-py) plots the confidence ellipsoids of a mixture of two Gaussians fitted with Expectation Maximisation (the GaussianMixture class) and Variational Inference (the BayesianGaussianMixture class). [How to select the K using bic](https://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_selection.html#sphx-glr-auto-examples-mixture-plot-gmm-selection-py) shows model selection with information-theory criteria, over both the covariance type and the number of components. [Density estimation for gmm - nice graph](https://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_pdf.html#sphx-glr-auto-examples-mixture-plot-gmm-pdf-py) plots the density estimate of a mixture of two Gaussians generated with different centers and covariance matrices.

### DBSCAN

Everything above assumes roughly round clusters around a center. DBSCAN drops that assumption and groups points by density. How to use effectively, and A practical guide to dbscan - pretty good, used to be the first two reads here; both are kept at the end of the page.

[a DBSCAN visualization - very good!](https://www.naftaliharris.com/blog/visualizing-dbscan-clustering/) is the interactive visualizing DBSCAN clustering page. [DBSCAN for GPS.](https://geoffboeing.com/2014/08/clustering-to-reduce-spatial-data-set-size/) is Geoff Boeing's post on clustering to reduce spatial data set size. Because scikit-learn's DBSCAN has no built-in function aside from fit\_predict to assign new points to existing clusters, while k-means has a predict method, [Custom DBSCAN “predict”](https://stackoverflow.com/questions/27822752/scikit-learn-predicting-new-points-with-dbscan) is slaw's question on how to write one. For geographic points, [Haversine distances for](https://kanoki.org/2019/12/27/how-to-calculate-distance-in-python-and-pandas-using-scipy-spatial-and-distance-functions/) is the kanoki guide to calculating distance in Python and Pandas with scipy spatial and distance functions: finding the distance between two coordinates or cities and generating a distance matrix.

Optimized dbscans come next, for when the plain version is too slow. muDBSCAN, paper - A fast, exact, and scalable algorithm for DBSCAN clustering. This repository contains the implementation for the distributed spatial clustering algorithm proposed in the paper μDBSCAN: An Exact Scalable DBSCAN Algorithm for Big Data Exploiting Spatial Locality; the muDBSCAN repository page is <https://githubmemory.com/repo/AdityaAS/MuDBSCAN>. [Dbscan multiplex](https://github.com/GGiecold/DBSCAN_multiplex) is a fast and memory-efficient implementation of DBSCAN (Density-Based Spatial Clustering of Applications with Noise). [Fast dbscan](https://github.com/harmslab/fast_dbscan) is a lightweight, fast dbscan implementation for use on peptide strings. It uses pure C for the distance calculations and clustering. This code is then wrapped in python. The [Faster dbscan paper](https://arxiv.org/pdf/1702.08607.pdf) is the arXiv paper on faster DB-scan and HDB-scan in low-dimensional Euclidean spaces.

### ST-DBSCAN

GPS points also carry time, and ST-DBSCAN adds that dimension. The [Paper - st-dbscan an algo for clustering spatio temporal data](https://www.sciencedirect.com/science/article/pii/S0169023X06000218) is on ScienceDirect. The [Popular git](https://github.com/eubr-bigsea/py-st-dbscan) is an implementation of the ST-DBScan algorithm in Python, and the other [git](https://github.com/gitAtila/ST-DBSCAN) implements ST-DBSCAN based on Birant 2007.

### HDBSCAN\*

DBSCAN needs one density threshold for the whole dataset; HDBSCAN makes it hierarchical so that choice goes away. The same notes are in [Anomaly Detection](/predictive-ml/anomaly-detection).

(what is?) HDBSCAN is a clustering algorithm developed by [Campello, Moulavi, and Sander](http://link.springer.com/chapter/10.1007%2F978-3-642-37456-2_14). It extends DBSCAN by converting it into a hierarchical clustering algorithm, and then using a technique to extract a flat clustering based in the stability of clusters.

The [Github code](https://github.com/scikit-learn-contrib/hdbscan) is a high performance implementation of HDBSCAN clustering. The (great) [Documentation](http://hdbscan.readthedocs.io/en/latest/basic_hdbscan.html) comes with examples, for clustering, outlier detection, comparison, benchmarking and analysis! The ([jupytr example](http://nbviewer.jupyter.org/github/scikit-learn-contrib/hdbscan/blob/master/notebooks/How%20HDBSCAN%20Works.ipynb)) explains the goal of finding islands of higher density amid a sea of sparser noise, because real data is messy and has outliers, corrupt data, and noise, and single linkage clustering on its own is sensitive to a noise point that bridges two islands. Take a look and see how to use it; usage examples are also in the docs and github.

What are the algorithm’s [steps](http://nbviewer.jupyter.org/github/scikit-learn-contrib/hdbscan/blob/master/notebooks/How%20HDBSCAN%20Works.ipynb):

1. Transform the space according to the density/sparsity.
2. Build the minimum spanning tree of the distance weighted graph.
3. Construct a cluster hierarchy of connected components.
4. Condense the cluster hierarchy based on minimum cluster size.
5. Extract the stable clusters from the condensed tree.

### OPTICS

OPTICS attacks the same varying-density problem by ordering points instead of building a hierarchy. ([What is?](https://en.wikipedia.org/wiki/OPTICS_algorithm)) Ordering points to identify the clustering structure (OPTICS) is an algorithm for finding density-based[\[1\]](https://en.wikipedia.org/wiki/OPTICS_algorithm#cite_note-1) [clusters](https://en.wikipedia.org/wiki/Cluster_analysis) in spatial data.

Its basic idea is similar to [DBSCAN](https://en.wikipedia.org/wiki/DBSCAN),[\[3\]](https://en.wikipedia.org/wiki/OPTICS_algorithm#cite_note-3) but it addresses one of DBSCAN's major weaknesses: the problem of detecting meaningful clusters in data of varying density. (How?) The points of the database are (linearly) ordered such that points which are spatially closest become neighbors in the ordering. Additionally, a special distance is stored for each point that represents the density that needs to be accepted for a cluster in order to have both points belong to the same cluster. (This is represented as a [dendrogram](https://en.wikipedia.org/wiki/Dendrogram).)

### SVM CLUSTERING

Density is one way to cluster without labels; another is to borrow a classifier and let it relabel the data until it settles. The same notes are in [Support vector clustering (SVC)](/predictive-ml/linear-separator-algorithms#support-vector-clustering-svc).

The [Paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2099486/) is Stephen Winters-Hilt's SVM clustering work, starting from the point that Support Vector Machines are a powerful method for classification (supervised learning) and asking how to use them for clustering (unsupervised learning). An SVM-based clustering algorithm is introduced that clusters data with no a priori knowledge of input classes.

1. The algorithm initializes by first running a binary SVM classifier against a data set with each vector in the set randomly labelled, this is repeated until an initial convergence occurs.
2. Once this initialization step is complete, the SVM confidence parameters for classification on each of the training instances can be accessed.
3. The lowest confidence data (e.g., the worst of the mislabelled data) then has its' labels switched to the other class label.
4. The SVM is then re-run on the data set (with partly re-labelled data) and is guaranteed to converge in this situation since it converged previously, and now it has fewer data points to carry with mislabelling penalties.
5. This approach appears to limit exposure to the local minima traps that can occur with other approaches. Thus, the algorithm then improves on its weakly convergent result by SVM re-training after each re-labeling on the worst of the misclassified vectors – i.e., those feature vectors with confidence factor values beyond some threshold.
6. The repetition of the above process improves the accuracy, here a measure of separability, until there are no misclassifications. Variations on this type of clustering approach are shown.

### COP-CLUSTERING

The last case is when some labels, or at least some pairwise hints, are available. The same notes are in [Semi Supervised](/problem-framing/semi-supervised).

Constrained K-means algorithm, [git](https://github.com/Behrouz-Babaki/COP-Kmeans), is a semi-supervised algorithm; the git link is a Python implementation of COP-KMEANS, and the [paper](https://web.cse.msu.edu/~cse802/notes/ConstrainedKmeans.pdf) is from the DaimlerChrysler Research and Technology Center, in the Proceedings of the Eighteenth International Conference on Machine Learning, 2001, p. 577–584. Its abstract:

Clustering is traditionally viewed as an unsupervised method for data analysis. However, in some cases information about the problem domain is available in addition to the data instances themselves. In this paper, we demonstrate how the popular k-means clustering algorithm can be profitably modified to make use of this information. In experiments with artificial constraints on six data sets, we observe improvements in clustering accuracy. We also apply this method to the real-world problem of automatically detecting road lanes from GPS data and observe dramatic increases in performance.

In the context of partitioning algorithms, instance level constraints are a useful way to express a priori knowledge about which instances should or should not be grouped together. Consequently, we consider two types of pairwise constraints: • Must-link constraints specify that two instances have to be in the same cluster. • Cannot-link constraints specify that two instances must not be placed in the same cluster.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-c6b162fe53699161b842cbfa1e7c8602c44574a8%2Fgimg-a3ece81709cd.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* thorough explanation 3. This address no longer opens: <https://towardsdatascience.com/introduction-to-k-nearest-neighbors-3b534bb11d26>
* Kernel Kmeans is part of TSLearn. This address no longer opens: <http://tslearn.readthedocs.io/en/latest/gen\\_modules/clustering/tslearn.clustering.GlobalAlignmentKernelKMeans.html>
* Elbow method. This address no longer opens: <https://blog.cambridgespark.com/how-to-determine-the-optimal-number-of-clusters-for-k-means-clustering-14f27070048f>
* elbow on medium using mean distance per cluster from the center. This address no longer opens: <https://towardsdatascience.com/what-is-k-ddf36926a752>
* finding the optimal K. This address no longer opens: <https://towardsdatascience.com/how-to-find-the-optimal-value-of-k-in-knn-35d936e554eb>
* How to use effectively. This address no longer opens: <https://towardsdatascience.com/how-to-use-dbscan-effectively-ed212c02e62>
* A practical guide to dbscan - pretty good. This address no longer opens: <https://towardsdatascience.com/a-practical-guide-to-dbscan-method-d4ec5ab2bc99>
* paper. This address no longer opens: <https://adityaas.github.io/>
* A good visual example of kmeans / gmm. This address no longer opens: <https://medium.com/sfu-cspmp/distilling-gaussian-mixture-models-701fa9546d9>


# Dimensionality Reduction Methods

High-dimensional data has to be projected down before it can be plotted, clustered, or fed to a smaller model, and each projection keeps something different. The page opens with overviews and the question of what is lost, then walks the linear methods (PCA, SVD, kernel PCA, LDA, KDA, ICA, LSA), moves to manifold methods such as t-SNE and IVIS, and ends with a note on principal component regression versus partial least squares.

The overviews that used to open this page, A series on DR for dummies on medium part 1 2 3, A small blog post about PCA, AE & TSNE in tensorflow, and Visualizing PCA/TSNE using plots, no longer open and are kept at the end of the page. The figure below is the overview picture that remains.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-07536fc17746ee20c9f45dfcf25f9b8dc39e5a20%2Fgimg-1c1a2fe7067c.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

Once data is projected, you want to look at it. [Parallex by uber for tsne \ pca visualization](https://github.com/uber-research/parallax) is Uber Research's tool for interactive embeddings visualization, the place to see what tsne / ae / pca did to the data. The next question is what the projection threw away: [Does dim-reduction loses information - yes and no, in pca yes only if you use less than the entire matrix](https://stats.stackexchange.com/questions/66060/does-dimension-reduction-always-lose-some-information) is the question of whether dimension reduction always loses information, for example whether very sparse data might have a better encoding where nothing is lost. And when speed matters, [Performance comparison between dim-reduction implementations, tsne etc.](https://umap-learn.readthedocs.io/en/latest/benchmarking.html) is the benchmark that goes with the figure below.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-de55446d64ac47fe5064395ab4542d36c35c3940%2Fgimg-ae6e068d52b7.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

### PCA

PCA is where every linear method starts, so its math comes first, then scaling it up, then using it correctly. The same notes are in [DIMENSIONALITY REDUCTION](#dimensionality-reduction) and [PCA for log anomaly detection](/predictive-ml/templatization#pca-for-log-anomaly-detection).

The Machine learning mastery sequence builds it from the ground up. [Expected value, variance, covariance ](https://machinelearningmastery.com/introduction-to-expected-value-variance-and-covariance)is the gentle NumPy introduction to those fundamental statistics, useful for a better understanding of your data. [PCA](https://machinelearningmastery.com/calculate-principal-component-analysis-scratch-python/) is the from-scratch calculation: **(remove the mean from A, calculate cov(A), calculate eig(cov), A\*eigK = PCA)**. [EigenDecomposition](https://machinelearningmastery.com/introduction-to-eigendecomposition-eigenvalues-and-eigenvectors/) explains **what is an eigen vector - simply put its a vector that satisfies A\*v = lambda\*v, how to use eig() and how to confirm an eigenvector/eigenvalue and reconstruct the original A matrix.** [SVD](https://machinelearningmastery.com/singular-value-decomposition-for-machine-learning) is how to calculate the SVD from scratch with Python, matrix decomposition (matrix factorization) as describing a matrix by its constituent elements. What is missing is how the EigenDecomposition is calculated.

The from-scratch version does not fit a very large matrix. [PCA on large matrices!](https://amedee.me/post/pca-large-matrices/) argues you don't need Spark: it starts from a PCA of a matrix with 1 million rows and 13,000 columns that took 27 hours on a Spark cluster with a 16 core machine and 30GB of RAM, and shows that PCA on several billion element matrices can be done much faster. The two tricks are Randomized svd and Incremental svd.

With the mechanics in hand, the tutorials show the whole flow. [PCA on Iris](http://sebastianraschka.com/Articles/2015_pca_in_3_steps.html) is Sebastian Raschka's step-by-step tutorial through standardization, covariance matrices, eigendecomposition, explained variance, and projection, with Python code. Three background reads are marked (did not read): [What is PCA?](https://stats.stackexchange.com/questions/222/what-are-principal-component-scores) asks what principal component scores are, [What is a covariance matrix?](https://en.wikipedia.org/wiki/Covariance_matrix) is the Wikipedia covariance matrix article, and [Variance covariance matrix](http://stattrek.com/matrix-algebra/covariance-matrix.aspx) shows how to use matrix methods to generate a variance-covariance matrix from a matrix of raw data, with a sample problem and solution. [Visualization of the first PCA vectors](https://medium.com/@rtjeannier/using-pca-to-visualize-high-dimensional-data-6ff028c911c5) plots the first vectors, though it is unclear what he is trying to show. [A very nice introductory tutorial on how to use PCA](https://www.analyticsvidhya.com/blog/2016/03/practical-guide-principal-component-analysis-python/) is Himanshi Singh's guide to how PCA helps with large, correlated datasets in data science projects.

For the full theory, **\*\*** An in-depth tutorial on PCA (paper) is kept at the end of the page, and the tutorial paper that remains is linked twice: **\*\*** [yet another tutorial paper on PCA (looks good)](https://www.researchgate.net/publication/309165405_Principal_component_analysis_-_a_tutorial), also noted as [Another tutorial paper - looks decent](https://www.researchgate.net/publication/309165405_Principal_component_analysis_-_a_tutorial). In a real pipeline, [How to use PCA in Cross validation and for train\test split](https://stats.stackexchange.com/questions/114560/pca-on-train-and-test-datasets-do-i-need-to-merge-them) settles where PCA is fitted (bottom line, do it on the train only.)

PCA also leads straight to whitening. [PCA whitening](http://mccormickml.com/2014/06/03/deep-learning-tutorial-pca-and-whitening/) is the deep learning tutorial on PCA and whitening, and the [Stanford tutorial](http://ufldl.stanford.edu/tutorial/unsupervised/PCAWhitening/) is the same topic in the Unsupervised Feature Learning and Deep Learning Tutorial. For (pca/zca whitening), [Stackoverflow (really good)](https://stats.stackexchange.com/questions/117427/what-is-the-difference-between-zca-whitening-and-pca-whitening/117459) is RockTheStar's question on the difference between ZCA whitening and normal PCA whitening (dividing principal components by the square roots of the PCA eigenvalues), where ZCA whitening is PCA whitening multiplied by the PCA eigenvectors, and what ZCA is used for.

There are two things we are trying to accomplish with whitening:

1. Make the features less correlated with one another.
2. Give all of the features the same variance.

Whitening has two simple steps:

1. Project the dataset onto the eigenvectors. This rotates the dataset so that there is no correlation between the components.
2. Normalize the the dataset to have a variance of 1 for all components. This is done by simply dividing each component by the square root of its eigenvalue.

### SVD

SVD is the decomposition under PCA, and it gets its own entry because it shows up again in recommenders. The same notes are in [Recommender Systems](/ai-product/recommender-systems).

An explanation about SVD’s formulas used to sit here; it is kept at the end of the page.

### KPCA

PCA is linear, so the next step is what happens when the structure is not. The slides behind this section (First they say that, kept at the end of the page) run in three steps. First they say that Autoencoder is PCA based on their equation, i.e. minimize the reconstruction error formula.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-bd6445eeed5f34bd3fa3aed6a138388247d92ffb%2Fgimg-9f08e204bf8d.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

Then they say that PCA cant separate certain non-linear situations (circle within a circle), therefore they introduce kernel based PCA (using the kernel trick - like svm) which mapps the space to another linearly separable space, and performs PCA on it.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-37f81c57b1dba7c59c7c7e419239b30bc8b607a5%2Fgimg-0c3115d49bb3.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-bf569b1afdbcf21f09cf9bb98479db5bd6f883a9%2Fgimg-4a5e1c55a298.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

Finally, they show results of how KPCA works well on noisy images, compared to PCA.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-9a8c7f84d9555aa23f3939d9534cd1ffda5cb4cc%2Fgimg-2c7ca7114b6b.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

### LDA - Linear discriminant analysis

PCA and KPCA ignore labels. When labels exist, LDA uses them, which makes it supervised dimensionality reduction. The same notes are in [Normalization & Scaling](/data/processing-intro/normalization-and-scaling).

[A comparison / tutorial with code on pca vs lda - read!](http://rstudio-pubs-static.s3.amazonaws.com/84669_cd15214061d44e1493ffee69c5d55925.html) is Zhao Hao's "How to properly perform LDA". A comprehensive tutorial on LDA - read! is kept at the end of the page. Dim reduction with LDA - nice examples.

([Not to be confused with the other LDA](http://sebastianraschka.com/Articles/2014_python_lda.html)) - Linear Discriminant Analysis (LDA) is most commonly used as dimensionality reduction technique in the pre-processing step for pattern-classification and machine learning applications. The goal is to project a dataset onto a lower-dimensional space with good class-separability in order avoid overfitting (“curse of dimensionality”) and also reduce computational costs.

PCA vs LDA:

Both Linear Discriminant Analysis (LDA) and Principal Component Analysis (PCA) are linear transformation techniques used for dimensionality reduction. PCA can be described as an “unsupervised” algorithm, since it “ignores” class labels and its goal is to find the directions (the so-called principal components) that maximize the variance in a dataset. In contrast to PCA, LDA is “supervised” and computes the directions (“linear discriminants”) that will represent the axes that maximize the separation between multiple classes.

Although it might sound intuitive that LDA is superior to PCA for a multi-class classification task where the class labels are known, this might not always the case. For example, comparisons between classification accuracies for image recognition after using PCA or LDA show that PCA tends to outperform LDA if the number of samples per class is relatively small ([PCA vs. LDA](http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=908974), A.M. Martinez et al., 2001). In practice, it is also not uncommon to use both LDA and PCA in combination.

Best Practice: PCA for dimensionality reduction can be followed by an LDA. But before we skip to the results of the respective linear transformations, let us quickly recapitulate the purposes of PCA and LDA: PCA finds the axes with maximum variance for the whole data set where LDA tries to find the axes for best class separability. In practice, often a LDA is done followed by a PCA for dimensionality reduction.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-a2259d4c3bc8a9527cc7aea7da654970a238e565%2Fgimg-5cf92386ed0c.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

\*\* To fully understand the details please follow the LDA link to the original and very informative article\*\*

\*\*\* TODO: need some benchmarking for PCA\LDA\LSA\ETC..\*\*

### KDA - KERNEL DISCRIMINANT ANALYSIS

Just as KPCA is the kernel version of PCA, KDA is the kernel version of discriminant analysis. The [pyDML package](https://pydml.readthedocs.io/en/latest/dml.html#dml.kda.KDA) has KDA. This package provides the classic algorithms of supervised distance metric learning, together with some of the newest proposals.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-095fe670bfb5045f58aea985f9989c86491c7c31%2Fgimg-5b71be433162.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

### ICA

LDA changes the goal from variance to class separation; ICA changes it from global directions to local, independent ones. While PCA is global, it finds global variables (with images we get eigen faces, good for reconstruction) that maximizes variance in orthogonal directions, and is not influenced by the TRANSPOSE of the data matrix. On the other hand, ICA is local and finds local variables (with images we get eyes ears, mouth, basically edges!, etc). ICA will result differently on TRANSPOSED matrices, unlike PCA, and its also “directional” - consider the “cocktail party” problem. On documents, ICA gives topics. It helps, similarly to PCA, to help us analyze our data.

Sparse [info on ICA with security returns.](https://www.quantopian.com/posts/an-experiment-with-independent-component-analysis)

### LSA

On documents, the linear method most people reach for is LSA, which is SVD applied to text. The same notes are in [LSA (TFIDF + SVD)](/natural-language-processing-nlp/topics-modeling#lsa-tfidf--svd).

[LSA](http://mccormickml.com/2016/03/25/lsa-for-text-classification-tutorial/) is quite simple, you just use SVD to perform dimensionality reduction on the tf-idf vectors–that’s really all there is to it! That line is from the latent semantic analysis for text classification tutorial with Python example code. And LSA CLUSTERING, the document clustering example, is kept at the end of the page.

Here is a very nice [tutorial about LSA,](https://technowiki.wordpress.com/2011/08/27/latent-semantic-analysis-lsa-tutorial/) with code, explaining what are the three matrices, word clustering, sentence clustering and vector importance. They say that for sentence space we need to remove the first vector as it is correlated with sentence length. The same text classification tutorial also shows how to [interpret LSA vectors](http://mccormickml.com/2016/03/25/lsa-for-text-classification-tutorial/).

PCA vs LSA: ([intuition1](https://stats.stackexchange.com/questions/65699/lsa-vs-pca-document-clustering), [intuition2](https://math.stackexchange.com/questions/3869/what-is-the-intuitive-relationship-between-svd-and-pca)). The first is a document clustering question that notes that PCA applies SVD to the term-covariance matrix while LSA applies it to the term-document matrix; the second asks what the intuitive relationship between SVD and PCA is, since both are eigenvalue methods that reduce a high-dimensional data set while retaining important information. What LSA buys you is reduction of the dimensionality, noise reduction, and incorporating relations between terms into the representation. SVD and PCA and "total least-squares" (and several other names) are the same thing. It computes the orthogonal transform that decorrelates the variables and keeps the ones with the largest variance. There are two numerical approaches: one by SVD of the (centered) data matrix, and one by Eigen decomposition of this matrix "squared" (covariance).

For text, LSA now competes with word embeddings. [LSA vs W2V](https://arxiv.org/pdf/1610.01520.pdf) is the comparative study of LSA vs Word2vec embeddings in small corpora, a case study in a dreams database.

### MANIFOLD

Every method so far is a linear projection, or a kernel trick on one. Manifold methods instead learn a curved high-to-low mapping, which is why they dominate visualization. [The best tutorial that explains manifold (high to low dim projection/mapping/visuzation)](https://jhui.github.io/2017/01/15/Machine-learning-Multi-dimensional-scaling-and-visualization/) is “Machine learning - Visualization, multi-dimensional scaling, Sammon mapping, IsoMap and t-sne”. [Many manifold methods used to visualize high dimensional data. ](http://scikit-learn.org/stable/modules/manifold.html#t-sne)is the scikit-learn manifold learning guide, the one that opens with "Look for the bare necessities" from Old Mother Nature’s recipes. [Comparing manifold methods](http://scikit-learn.org/stable/auto_examples/manifold/plot_compare_methods.html#sphx-glr-auto-examples-manifold-plot-compare-methods-py) is the illustration of dimensionality reduction on the S-curve dataset with various manifold learning methods.

#### Sammons embedding mapping

The first manifold method in that tutorial is Sammon mapping, and [In tensorflow](https://datawarrior.wordpress.com/2017/06/01/sammon-embedding-with-tensorflow/) is the Sammon Embedding with Tensorflow post from Everything about Data Analytics.

#### T-SNE

t-SNE is the manifold method this page spends the most time on, and the reading list carries its own warning about misreading it. The <https://towardsdatascience.com/t-sne-python-example-1ded9953f26> post is the code and in-depth tutorial on TSNE, mapping probabilities to distributions\*\*\*\*\*\*\*\*. A great example of using PCA and then TSNE to see clusters that arent visible with PCA only is kept at the end of the page. [Misreading T-SNE](https://distill.pub/2016/misread-tsne/) is Distill's "How to Use t-SNE Effectively": t-SNE is extremely useful for visualizing high-dimensional data, but its plots can sometimes be mysterious or misleading, so this is a very important read.

In contrary to what it says on sklearn’s website, TSNE is not suited ONLY for visualization, you [can also use it for data reduction](https://lvdmaaten.github.io/tsne/). “t-Distributed Stochastic Neighbor Embedding (t-SNE) is a ([prize-winning](http://blog.kaggle.com/2012/11/02/t-distributed-stochastic-neighbor-embedding-wins-merck-viz-challenge/)) technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets.”

Comparing PCA and TSNE, then pushing PCA to TSNE and seeing what happens (as recommended in SKLEARN), used to be linked here and is kept at the end of the page. A TSNE + AUTOENCODER example is the other combination worth trying.

### TSNE

After the reading list, the intuition. [Stat quest](https://www.youtube.com/watch?v=NEaUSP4YerM\&list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF\&index=30) gives the jist of it: we assume a t- distribution on distances and remove those that are farther, normalized for density. T-dist used so that clusters are not clamped in the middle. The figures below draw that idea.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-b19623857e33f747ce2462239d36f9f670c8098b%2Fgimg-33abbc7a1d93.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-c4da475833d857c844645499841e5e77b3b79b5d%2Fgimg-6b3d7ed51c31.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

Iteratively moving from the left to the right:

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-d7350735883202329f662e12090ca9df5e8067d4%2Fgimg-19902d410251.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

[TSNE algorithm](https://www.oreilly.com/learning/an-illustrated-introduction-to-the-t-sne-algorithm) is Cyrille Rossant's illustrated introduction to t-distributed stochastic neighbor embedding. The caveats versus PCA come next. [Are there cases where PCA more suitable than TSNE](https://stats.stackexchange.com/questions/238538/are-there-cases-where-pca-is-more-suitable-than-t-sne?rq=1) starts from someone who ran PCA on 7 correlated measures of text correction behaviour and was asked why not t-SNE. [PCA preserving pairwise distances over tSNE?](https://stats.stackexchange.com/questions/176672/what-is-meant-by-pca-preserving-only-large-pairwise-distances) asks what it means that PCA only preserves large pairwise distances, so points far apart in high dimensions also appear far apart in the projection. How why, all here. Another advice about using tsne and the possible misinterpetations is <https://towardsdatascience.com/why-you-are-using-t-sne-wrong-502412aab0c0>.

#### IVIS

When the dataset is too large for t-SNE, IVIS is a learned alternative. The [Paper: ](https://www.nature.com/articles/s41598-019-45301-0)is "Structure-preserving visualisation of high dimensional single-cell datasets" in Scientific Reports. The [Git](https://github.com/beringresearch/ivis) repository is dimensionality reduction in very large datasets using Siamese Networks, and the [docs](https://bering-ivis.readthedocs.io/en/latest/) are the ivis dimensionality reduction documentation. The companion repositories are [Ivis animate](https://github.com/beringresearch/ivis-animate), beringresearch/ivis-animate, and [Ivis explain](https://github.com/beringresearch/ivis-explain), beringresearch/ivis-explain.

#### DIMENSIONALITY REDUCTION

The page ends where dimensionality reduction meets regression: reducing the features and then fitting a model on them. The same notes are in [PCA](#pca).

PRINCIPAL COMPONENT REGRESSION (PCR) / PARTIAL LEAST SQUARES (PLS)

[Principal component regression (PCR) Partial least squares and (PLS)](https://www.kdnuggets.com/2017/11/10-statistical-techniques-data-scientists-need-master.html/2) - basically PCA and linear regression, however PLS makes use of the response variable in order to identify the new features.

One can describe Principal Components Regression as an approach for deriving a low-dimensional set of features from a large set of variables. The first principal component direction of the data is along which the observations vary the most. In other words, the first PC is a line that fits as close as possible to the data. One can fit p distinct principal components. The second PC is a linear combination of the variables that is uncorrelated with the first PC, and has the largest variance subject to this constraint. The idea is that the principal components capture the most variance in the data using linear combinations of the data in subsequently orthogonal directions. In this way, we can also combine the effects of correlated variables to get more information out of the available data, whereas in regular least squares we would have to discard one of the correlated variables.

The PCR method that we described above involves identifying linear combinations of X that best represent the predictors. These combinations (directions) are identified in an unsupervised way, since the response Y is not used to help determine the principal component directions. That is, the response Y does not supervise the identification of the principal components, thus there is no guarantee that the directions that best explain the predictors also are the best for predicting the response (even though that is often assumed). Partial least squares (PLS) are a supervised alternative to PCR. Like PCR, PLS is a dimension reduction method, which first identifies a new smaller set of features that are linear combinations of the original features, then fits a linear model via least squares to the new M features. Yet, unlike PCR, PLS makes use of the response variable in order to identify the new features.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* 1. This address no longer opens: <https://towardsdatascience.com/https-medium-com-abdullatif-h-dimensionality-reduction-for-dummies-part-1-a8c9ec7b7e79?\\_branch\\_match\\_id=584170448791192656>
* 2. This address no longer opens: <https://towardsdatascience.com/dimensionality-reduction-for-dummies-part-2-3b1e3490bdc9>
* 3. This address no longer opens: <https://towardsdatascience.com/dimensionality-reduction-for-dummies-part-3-f25729f74c0a>
* A small blog post about PCA, AE & TSNE. This address no longer opens: <https://towardsdatascience.com/reducing-dimensionality-from-dimensionality-reduction-techniques-f658aec24dfe>
* Visualizing PCA/TSNE using plots. This address no longer opens: <https://towardsdatascience.com/visualising-high-dimensional-datasets-using-pca-and-t-sne-in-python-8ef87e7915b>
* An in-depth tutorial on PCA (paper). This address no longer opens: <https://www.cs.princeton.edu/picasso/mats/PCA-Tutorial-Intuition\\_jp.pdf>
* An explanation about SVD’s formulas.. This address no longer opens: <https://blog.statsbot.co/singular-value-decomposition-tutorial-52c695315254>
* First they say that. This address no longer opens: <https://web.cs.hacettepe.edu.tr/\\~aykut/classes/fall2016/bbm406/slides/l25-kernel\\_pca.pdf>
* A comprehensive tutorial on LDA - read!. This address no longer opens: <https://iksinc.online/2018/11/12/dimensionality-reduction-via-linear-discriminant-analysis/>
* LSA CLUSTERING. This address no longer opens: <http://mccormickml.com/2015/08/05/document-clustering-example-in-scikit-learn/>
* A great example of using PCA and then TSNE to see clusters that arent visible with PCA only.. This address no longer opens: <https://towardsdatascience.com/dimensionality-reduction-by-stacking-pca-and-t-sne-420d9fcfab54>
* Comparing PCA and TSNE, then pushing PCA to TSNE and seeing what happens (as recommended in SKLEARN. This address no longer opens: <https://medium.com/@luckylwk/visualising-high-dimensional-datasets-using-pca-and-t-sne-in-python-8ef87e7915b>


# Anomaly Detection

Anomaly detection asks whether a new point looks like the data you already have, and the first decision is whether you are doing novelty detection or outlier detection. The page defines that split, collects the general reading and the libraries, then walks the methods from elliptic envelopes and LOF through isolation forests and one-class SVM, and ends with clustering metrics such as silhouette for related checks.

The same notes are in [ANOMALY DETECTION TS](/predictive-ml/time-series-search#anomaly-detection-ts), [Drift](/ai-engineering/mlops-monitoring-and-alerts#drift), [Fraud Detection](/ai-product/fraud-detection), [HDBSCAN\*](/predictive-ml/clustering-algorithms#hdbscan), and [PCA for log anomaly detection](/predictive-ml/templatization#pca-for-log-anomaly-detection).

The task is to decide “whether a new observation belongs to the same distribution as existing observations (it is an inlier), or should be considered as different (it is an outlier). Often, this ability is used to clean real data sets.

Two important distinctions must be made:

| novelty detection: |                                                                                                                                  |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------- |
|                    | The training data is not polluted by outliers, and we are interested in detecting anomalies in new observations.                 |
| outlier detection: |                                                                                                                                  |
|                    | The training data contains outliers, and we need to fit the central mode of the training data, ignoring the deviant observations |

With that split in mind, the general reading starts simple. Using IQR for AD and why IQR difference is 2.7 sigma is explained in <https://towardsdatascience.com/why-1-5-in-iqr-method-of-outlier-detection-5d07fdc82097>. Medium — good: <https://towardsdatascience.com/anomaly-detection-for-dummies-15f148e559c1> is Julia Bohutska's "Anomaly Detection - How to Tell Good Performance from Bad". The [kdnuggets](https://www.kdnuggets.com/2017/04/datascience-introduction-anomaly-detection.html) piece is its Introduction to Anomaly Detection. For [Z-score and other moving averages.](https://turi.com/learn/userguide/anomaly_detection/moving_zscore.html), the link now lands on the apple/turicreate repository; Turi Create simplifies the development of custom machine learning models. A survey that sat here is kept at the end of the page.

From reading to code, [A great tutorial](https://www.analyticsvidhya.com/blog/2019/02/outlier-detection-python-pyod/) is an awesome tutorial on what an outlier is and how to use the PyOD library for outlier detection in Python. It is about AD using 20 algos in a [single python package](https://github.com/yzhao062/pyod); that repository now describes itself as a Python library for anomaly detection across tabular, time series, graph, text, image, and audio data, with 60+ detectors. For rare events in sequences, [Mastery on classifying rare events using lstm-autoencoder](https://machinelearningmastery.com/lstm-model-architecture-for-rare-event-time-series-forecasting/) is MachineLearningMastery's LSTM model architecture for rare event time series forecasting.

The same notes are in [AUTOENCODERS](/deep-learning/autoencoders#autoencoders) and [Timeseries](/predictive-ml/forecasting).

Before picking one method, it helps to see them side by side. A [comparison](http://scikit-learn.org/stable/modules/outlier_detection.html#outlier-detection) of One-class SVM versus Elliptic Envelope versus Isolation Forest versus LOF in sklearn is part of the novelty and outlier detection guide. (The examples below illustrate how the performance of the [covariance.EllipticEnvelope](http://scikit-learn.org/stable/modules/generated/sklearn.covariance.EllipticEnvelope.html#sklearn.covariance.EllipticEnvelope) degrades as the data is less and less unimodal. The [svm.OneClassSVM](http://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html#sklearn.svm.OneClassSVM) works better on data with multiple modes and [ensemble.IsolationForest](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html#sklearn.ensemble.IsolationForest) and [neighbors.LocalOutlierFactor](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.LocalOutlierFactor.html#sklearn.neighbors.LocalOutlierFactor) perform well in every cases.) Each of those four API pages links to the gallery examples that compare anomaly detection algorithms for outlier detection on toy datasets.

The applied examples come next. [Using Autoencoders](https://shiring.github.io/machine_learning/2017/05/01/fraud) — the information is there, but its all over the place. Twitter anomaly is another reference. Microsoft anomaly — a well documented black box, i cant find a description of the algorithm, just hints to what they sort of did. Its blog post, [up/down trend, dynamic range, tips and dips](https://blogs.technet.microsoft.com/machinelearning/2014/11/05/anomaly-detection-using-machine-learning-to-detect-abnormalities-in-time-series-data/), is "Anomaly Detection – Using Machine Learning to Detect Abnormalities in Time Series Data". The [Api here](https://docs.microsoft.com/en-us/azure/machine-learning/team-data-science-process/apps-anomaly-detection-api) link now lands on Microsoft's Cloud Adoption Framework page on the process to plan for AI adoption. STL and [LSTM for anomaly prediction](https://github.com/omri374/moda/blob/master/moda/example/lstm/LSTM_AD.ipynb) is the notebook in omri374/moda, a models and evaluation framework for trending topics detection.

Two more Medium posts round out the reading. Medium on AD is <https://towardsdatascience.com/machine-learning-for-anomaly-detection-and-condition-monitoring-d4614e7de770> on Towards Data Science, and Medium on AD using mahalanobis, AE and is <https://towardsdatascience.com/how-to-use-machine-learning-for-anomaly-detection-and-condition-monitoring-6742f82900d7>.

### OUTLIER DETECTION

Once the problem is framed, the next question is which library to run it with. [Alibi Detect](https://github.com/SeldonIO/alibi-detect) is an open source Python library focused on outlier, adversarial and drift detection. The package aims to cover both online and offline detectors for tabular data, text, images and time series. The outlier detection methods should allow the user to identify global, contextual and collective outliers.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-2edc2adcdc56286d7b742e1a3e87c4ff63692d11%2Fgimg-c9ce572f20fe.png?alt=media" alt=""><figcaption><p>Alibi Detect.</p></figcaption></figure>

The PyOD package from the tutorial above has its own documentation, the [Pyod](https://pyod.readthedocs.io/en/latest/) pyod 3.6.6 documentation, and the three figures below are from PyOD.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-03f13644865a24f3a9f5556bf73c8e63b3afb665%2Fgimg-4462e0610ead.png?alt=media" alt=""><figcaption><p>PyOD.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-834ca8a1bce254db3787ed84f7307d41afc4055f%2Fgimg-79ab319d3e40.png?alt=media" alt=""><figcaption><p>PyOD.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-d9c86b1efb6ffd306aab7d0c3d0338cdf0cf4b07%2Fgimg-0fdcf6a17ed3.png?alt=media" alt=""><figcaption><p>PyOD.</p></figcaption></figure>

From the same author, the (great) [Anomaly detection resources](https://github.com/yzhao062/anomaly-detection-resources) list collects anomaly detection related books, papers, videos, and toolboxes, last updated in late 2025 for LLM and VLM works. [Novelty and outlier detection inm sklearn](https://scikit-learn.org/stable/modules/outlier_detection.html) is the scikit-learn guide for applications that need to decide whether a new observation is an inlier or an outlier.

When one detector is not enough, [SUOD](https://github.com/yzhao062/suod) (Scalable Unsupervised Outlier Detection) is an acceleration framework for large-scale unsupervised outlier detector training and prediction. Notably, anomaly detection is often formulated as an unsupervised problem since the ground truth is expensive to acquire. To compensate for the unstable nature of unsupervised algorithms, practitioners often build a large number of models for further combination and analysis, e.g., taking the average or majority vote. However, this poses scalability challenges in high-dimensional, large datasets, especially for proximity-base models operating in Euclidean space.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-bed25742c8535df4a115bbe0ef5107e33dccf50e%2Fgimg-8bb2d61f1082.png?alt=media" alt=""><figcaption><p>SUOD.</p></figcaption></figure>

SUOD is therefore proposed to address the challenge at three complementary levels: random projection (data level), pseudo-supervised approximation (model level), and balanced parallel scheduling (system level). As mentioned, the key focus is to accelerate the training and prediction when a large number of anomaly detectors are presented, while preserving the prediction capacity. Since its inception in Jan 2019, SUOD has been successfully used in various academic researches and industry applications, include PyOD [\[2\]](https://github.com/yzhao062/suod#zhao2019pyod) and [IQVIA](https://www.iqvia.com/) medical claim analysis. It could be especially useful for outlier ensembles that rely on a large number of base estimators.

For time series metrics in production, [Skyline](https://github.com/earthgecko/skyline) is real-time anomaly detection for time series metrics: 11 years of production anomaly detection evolution, multi-algorithm ensembles, pattern recognition using semi-supervised and unsupervised learning, and correlation analysis, learning and getting better over time with no GPU required, so it can run on a VPS.

Scikit-lego outliers is the last library here; its page is kept at the end, and the figures below are from it.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-28eecd972ee648b012e7bb67305d3f0d7174d3db%2Fgimg-a2e83f028e2a.png?alt=media" alt=""><figcaption><p>Scikit-lego outliers.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-0facfba7ab6a629f1b87645d2cd19a1ab5fd1d36%2Fgimg-d3e77feb78ab.png?alt=media" alt=""><figcaption><p>Scikit-lego outliers.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-ff98b9edcf4ec2a24a4d19a857c4dd8804d0712f%2Fgimg-c69974b4f0f1.png?alt=media" alt=""><figcaption><p>Scikit-lego outliers.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-1d2945c6f24663961a23eb6516239f2932659284%2Fgimg-248cf19aa992.png?alt=media" alt=""><figcaption><p>Scikit-lego outliers.</p></figcaption></figure>

### ELLIPTIC ENVELOPE

The first method in the sklearn comparison is the one with the strongest assumption. We assume that the regular data come from a known distribution (e.g. data are Gaussian distributed). From this assumption, we generally try to define the “shape” of the data, and can define outlying observations as observations which stand far enough from the fit shape. That is also why it degrades as the data becomes less unimodal.

### LOCAL OUTLIER FACTOR

When no single distribution fits, LOF compares each point only to its neighbours. [LOF](http://scikit-learn.org/stable/modules/outlier_detection.html#local-outlier-factor) computes a score (called local outlier factor) reflecting the degree of abnormality of the observations. It measures the local density deviation of a given data point with respect to its neighbors. The idea is to detect the samples that have a substantially lower density than their neighbors. In practice the local density is obtained from the k-nearest neighbors.

The LOF score of an observation is equal to the ratio of the average local density of his k-nearest neighbors, and its own local density: a normal instance is expected to have a local density similar to that of its neighbors, while abnormal data are expected to have much smaller local density.

### ISOLATION FOREST

LOF measures density; an isolation forest measures how easy a point is to cut off. The best resource to explain isolation forest (kept at the end of the page) gives the basic idea: for an anomaly (in the example) only 4 partitions are needed, for a regular point in the middle of a distribution, you need many many more.

The scikit-learn [Isolation Forest](http://scikit-learn.org/stable/auto_examples/ensemble/plot_isolation_forest.html) example describes the model as an ensemble of "Isolation Trees" that "isolate" observations by recursive random partitioning. Isolating observations:

* randomly selecting a feature
* randomly selecting a split value between the maximum and minimum values of the selected feature.

Recursive partitioning can be represented by a tree structure, the number of splittings required to isolate a sample is equivalent to the path length from the root node to the terminating node. This path length, averaged over a forest of such random trees, is a measure of normality and our decision function. Random partitioning produces noticeable shorter paths for anomalies. So when a forest of random trees collectively produce shorter path lengths for particular samples, they are highly likely to be anomalies.

[the paper is pretty good too -](https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/icdm08b.pdf) In the training stage, iTrees are constructed by recursively partitioning the given training set until instances are isolated or a specific tree height is reached of which results a partial model.

Note that the tree height limit l is automatically set by the sub-sampling size ψ: l = ceiling(log2 ψ), which is approximately the average tree height \[7].

The rationale of growing trees up to the average tree height is that we are only interested in data points that have shorter-than average path lengths, as those points are more likely to be anomalies

### ONE CLASS SVM

The last method draws a boundary around the normal data instead of scoring each point, which is why the sklearn comparison found it better on data with multiple modes. The same notes are in [SUPPORT VECTOR MACHINES (SVM)](/predictive-ml/linear-separator-algorithms#support-vector-machines-svm).

A nice article about ocs, with github code, two methods are described; it is kept at the end of the page. The [Resources for ocsvm](https://www.quora.com/What-is-a-good-resource-for-understanding-One-Class-SVM-for-distribution-esitmation) thread asks what a good resource is for understanding One Class SVM for distribution estimation. It looks like there are two such methods — The 2nd one: The algorithm obtains a spherical boundary, in feature space, around the data. The volume of this hypersphere is minimized, to minimize the effect of incorporating outliers in the solution.

The resulting hypersphere is characterized by a center and a radius R>0 as distance from the center to (any support vector on) the boundary, of which the volume R2 will be minimized.

### CLUSTERING METRICS

Anomalies and clusters are two sides of the same question, so the page ends with how to judge a clustering, for community detection, text clusters, etc. The same notes are in [Clustering Algorithms](/predictive-ml/clustering-algorithms).

For word embeddings in particular, the [Google search for convenience](https://www.google.com/search?biw=1600\&bih=912\&sxsrf=ALeKk00NbB52pfM6J1N42ieEddIOirBmcQ%3A1597514997743\&ei=9SQ4X-_uLLLhkgWJsIbADw\&q=word+embedding+silhouette+score\&oq=word+embedding+silhouette+score\&gs_lcp=CgZwc3ktYWIQAzoECAAQRzoECCMQJzoHCCMQsAIQJ1DVd1jqjQFgm5ABaARwAXgBgAGMAogBrQ2SAQUwLjkuMpgBAKABAaoBB2d3cy13aXrAAQE\&sclient=psy-ab\&ved=0ahUKEwivvdqP553rAhWysKQKHQmYAfg4ChDh1QMIDA\&uact=5) is a saved Google Search for word embedding silhouette score.

**Silhouette:**

[TFIDF, PCA, SILHOUETTE](https://medium.com/data-science/mmmm-foodporn-a-clustering-and-classification-study-using-natural-language-processing-e2eae8ddefe1) is the study for deciding how many clusters to use, the knee/elbow method; the same post is also at <https://towardsdatascience.com/mmmm-foodporn-a-clustering-and-classification-study-using-natural-language-processing-e2eae8ddefe1>, Dan Wilentz's "Mmmm Foodporn! A Clustering and Classification Study using Natural Language Processing", using data science to analyze cuisine popularity on Reddit. Its [Topic modelling clustering, cant access this document on github](https://github.com/danielwilentz/Cuisine-Classifier/blob/master/topic_modeling/clustering.ipynb) notebook uses unsupervised learning and NLP to cluster titles from posts on r/foodporn and classify new titles. [Embedding based silhouette community detection](https://link.springer.com/article/10.1007/s10994-020-05882-8#Sec10) applies silhouette to community detection. [A notebook](https://rlbarter.github.io/superheat-examples/word2vec/), using the SuperHeat package, clusters a w2v cosine similarity matrix, measuring using silhouette score.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* A survey. This address no longer opens: <https://d1wqtxts1xzle7.cloudfront.net/49916547/Mohiuddin\\_Survey\\_financial\\_2015.pdf?1477591055=\\&response-content-disposition=inline%3B+filename%3DA\\_survey\\_of\\_anomaly\\_detection\\_techniques.pdf\\&Expires=1594649751\\&Signature=U~~N32meGWYyIIQz1zRYC4s2tCb7e5ut28GIBC3GSG4250UjhgTMQwEIB63zwPKtS5JyKew7RWVog8gytIhc3GSSfTwsRM7lqyghuDgbds-QMp3mNyVw2bYNztnoOWncHG8rhtkwUK1EbWcYeLKvqARnJoAS177C8r1GAhfKp14GgJzHpmnsoSkB6AowJ68nauf2VyA1b~~w1m\\~UfSNoWtjbL59clAqHn7nfqw5PGBuLHSSSxCa5PX09mADy4VzuOySzYjIviRwOlgT1eQrART0KqozqVSiGKM3SeapuI3K5tSERVPPSTnpupp--WJyYCNzzvPrdjB121P2XU7fq73wQ\\_\\_\\&Key-Pair-Id=APKAJLOHF5GGSLRBV4ZA>
* Pyod. This address no longer opens: <https://pyod.readthedocs.io/en/latest/pyod.html>
* Scikit-lego. This address no longer opens: <https://scikit-lego.readthedocs.io/en/latest/outliers.html>
* The best resource to explain isolation forest. This address no longer opens: <http://blog.easysol.net/using-isolation-forests-anamoly-detection/>
* A nice article about ocs, with github code, two methods are described. This address no longer opens: <http://rvlasveld.github.io/blog/2013/07/12/introduction-to-one-class-support-vector-machines/>
* two such methods. This address no longer opens: <http://rvlasveld.github.io/blog/2013/07/12/introduction-to-one-class-support-vector-machines/>


# Process Mining

A company writes down a process, but the documented process is often incomplete or ignored, and the real one only shows up in the event logs. Process mining recovers that real process from the logs. The page starts with how the process is modeled and checked for conformance, then moves to healthcare tutorials and a YouTube playlist, and ends with the tools and miners that do the work.

Processes were usually manual, giving trust in people following them, e.g. Process that was defined by the company, and no automation. Process mining identifies the process from the logs (**process discovery**).

The discovered process needs a notation. Modeling is done with [BPMN](https://www.bpmn.org/), the business process model notation language (UML) (DAG), whose official resource covers BPMN 2.0. BPMN is static, boxes & arrows. The alternative is the [Petri net](https://en.wikipedia.org/wiki/Petri_net), and the Cyber Physics [YouTube](https://www.youtube.com/watch?v=EmYVZuczJ6k) video on Petri nets walks through it. A Petri net is dynamic, token based, which allows simulations.

Once there is a model, conformance checking is a comparison between the real process and the discovered one. If you do not do certain parts in the process you are not compliant. For example, it can find out whether people are taking shortcuts: are they optimizing the process without knowing? The same comparison can be used for offline vs real time process bug alerting. The standardized log format behind all of this is XES.

### Tutorials

The ideas above become concrete in a four-part tutorial series, Process Mining with Python tutorial: A healthcare application. <https://medium.com/@c3_62722/process-mining-with-python-tutorial-a-healthcare-application-part-1-ae02027a050> is Part 1, the first article of the series, which lays out the parts that follow. <https://medium.com/@c3_62722/process-mining-with-python-tutorial-a-healthcare-application-part-2-4cf57053421f> is Part 2, <https://medium.com/@c3_62722/process-mining-with-python-tutorial-a-healthcare-application-part-3-cc9af986c122> is Part 3, and <https://medium.com/@c3_62722/process-mining-with-python-tutorial-a-healthcare-application-part-4-912286ee51b> is Part 4.

For video, YouTube has the pm4py tutorials playlist by Process Mining for Python, starting with tutorial #1: What is Process Mining? <https://www.youtube.com/watch?v=XLHtvt36g6U&list=PLkWuoFn9UEb5l41T4CMKPYHyRcL5ojI9Z>

### Tools

The tutorials assume the logs already exist, so the first requirement on tooling is that services need to be process-aware, i.e. send standardized logs, the setting <https://www.celonis.com/> is built around.

The algorithms that discover the process from those logs can deal with parallelism. The first is the [Alpha miner](http://mlwiki.org/index.php/Alpha_Algorithm), described on the mlwiki page <http://mlwiki.org/index.php/Alpha_Algorithm>. The second is the Inductive miner.

Those miners ship inside products. Process Intelligence Solutions GmbH (P.I.S.) develops process mining software including PM4Py and PMTk, aimed at helping organizations streamline their processes: <https://processintelligence.solutions/pm4py>. Celonis is the commercial platform: <https://www.celonis.com/?>. IBM Process Mining extracts process data from the business, identifies automation opportunities, prioritizes them by impact, and fast-tracks implementation: <https://www.ibm.com/products/process-mining>. To learn the fundamentals from the same vendor, Celonis has a video series on the fundamentals of process mining: <https://www.celonis.com/wils-process-mining-class/?>

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* <https://pm4py.fit.fraunhofer.de/static/assets/api/2.7.5.1/getting\\_started.html#understanding-process-mining>. This address no longer opens: <https://pm4py.fit.fraunhofer.de/static/assets/api/2.7.5.1/getting\\_started.html#understanding-process-mining>
* <https://pm4py.fit.fraunhofer.de/>. This address no longer opens: <https://pm4py.fit.fraunhofer.de/>


# Log Parsing / Templatization

Raw logs are free text, and before any anomaly detection can run on them each line has to be reduced to a template plus its variable parts. The page starts with the parsing problem and the Logpai collection, then moves through CRF-based templatization, Log3C and its sibling tools, and the LogParser benchmark, and ends with word vectors on logs, feature extraction windows, and PCA-based anomaly detection.

The same notes are in [ASSOCIATION RULES](/predictive-ml/data-mining#association-rules).

The entry point is [Awesome log analysis](https://github.com/logpai/awesome-log-analysis), logpai's list of awesome research on log analysis, anomaly detection, fault localization, and AIOps. The figure below is taken from that list.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-719abec8da5619b4a6f5f034cda72ed19c4831ae%2Fgimg-994a97533e75.png?alt=media" alt=""><figcaption><p>Awesome log analysis.</p></figcaption></figure>

The same repository keeps a papers.md file, (really good) [And list of papers for each field](https://github.com/logpai/awesome-log-analysis/blob/master/papers.md#anomaly-detection), which opens at the anomaly-detection papers. For a wider view, [How to use log analytics to detect log anomaly](https://www.msystechnologies.com/blog/how-to-use-log-analytics-to-detect-log-anomaly/) is more of a survey into the technologies available, and its figure follows.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-d3fe4cdb9068cceab2b9341112478b32d00f317e%2Fgimg-ba826d62f492.png?alt=media" alt=""><figcaption><p>How to use log analytics to detect log anomaly.</p></figcaption></figure>

### Logpai

Most of the tools on this page come from one group. [Logpai](https://github.com/logpai) is LOGPAI, Log Analytics Powered by AI, the GitHub organization that hosts the repositories below. Its [Loghub datasets](https://github.com/logpai/loghub) are a large collection of system log datasets for AI-driven log analytics \[ISSRE'23]. The [logpaI loglizer:](https://github.com/PinjiaHe) link points to the GitHub profile of PinjiaHe, whose paper [An Evaluation Study on Log Parsing and Its Use in Log Mining](https://jiemingzhu.github.io/pub/pjhe_dsn2016.pdf) comes out of the Chinese University of Hong Kong, Shenzhen. The loglizer code itself is in the [git](https://github.com/logpai/loglizer) repo, a machine learning toolkit for log-based anomaly detection \[ISSRE'16].

## CRF for templatization

Logpai's parsers are rule and clustering based, but templatization can also be framed as a sequence-labeling task. The same notes are in [CONDITIONAL RANDOM FIELDS (CRF)](/predictive-ml/probabilistic-models#conditional-random-fields-crf).

[Towards an NLP based log template generation algorithm for system log analysis](http://www.3at.work/papers/cfi2014.pdf) starts from system logs of network equipment, one of the most important sources for network management, where generating log templates (the meta format) from real log messages is still difficult. Its answer — CRF for templatization, i.e. NER style. The catch is the amount of training data:

> we can see that the more learning data given, the more accurately CRF produces log templates. Clearly a sufficient number of train data enables us to analyze less frequently appearing log templates. Therefore, it is reasonable that a log template can be analyzed correctly if train data include some of similar templates. However, in terms of log template accuracy, CRF requires 10000 train data to achieve same accuracy as Vaarandi’s algorithm

The three figures below are from the same paper.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-fc16b10ff4cfafc32205d2fc6b850428df8d106a%2Fgimg-ee74692bf5f3.png?alt=media" alt=""><figcaption><p>Towards an NLP based log template generation algorithm for system log analysis.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-3b30b6bbac74b8c4ec0364ba69857bcf5a19b0b5%2Fgimg-6f6397584cca.png?alt=media" alt=""><figcaption><p>Towards an NLP based log template generation algorithm for system log analysis.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-ab49780c6ba0b61c44270cc3063e2175746be274%2Fgimg-7a4c2aac8bf2.png?alt=media" alt=""><figcaption><p>Towards an NLP based log template generation algorithm for system log analysis.</p></figcaption></figure>

## Log3C

Parsing is only the first step; the point is to find the problems the logs describe. [log3C](https://github.com/logpai/Log3C) is logpai's log-based impactful problem identification using machine learning \[FSE'18], and the [paper](https://dl.acm.org/citation.cfm?id=3236083) is its ACM publication. Log3C is a general framework that identifies service system problems from system logs. It utilizes both system logs and system KPI metrics to promptly and precisely identify impactful system problems. Log3C consists of four steps: Log parsing, Sequence vectorization, Cascading Clustering and Correlation analysis. This is a joint work by CUHK and Microsoft Research. The repository contains the source code of Log3C, including data loading, sequence vectorization, cascading clustering, data saving, etc. The core part is the cascading clustering algorithm, which groups a large number of sequence vectors into clusters by iteratively sampling, clustering, matching. The three Log3C figures show that pipeline.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-a2bc681a639987beac1e6f8cd596872300448b9f%2Fgimg-c727546f3ae9.png?alt=media" alt=""><figcaption><p>Log3C.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-62d42c728a0d8ab9a6b54e45bbb1d80053fb8f91%2Fgimg-35d4d4b13787.png?alt=media" alt=""><figcaption><p>Log3C.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-4eac6896044273fdaf72be9cc610a4a8462bc97d%2Fgimg-92469c62bd66.png?alt=media" alt=""><figcaption><p>Log3C.</p></figcaption></figure>

The paper is explicit about its limits. Selection of KPI: In our experiments, we use failure rate as the KPI for problem identification. failure rate is an important KPI for evaluating system service availability. There are also other KPIs such as mean time between failures, average request latency, throughput, etc. In our future work, we will experiment with problem identification concerning different KPI metrics. Noises in labeling: Our experiments are based on three datasets that are collected as a period of logs on three different days. The engineers manually inspected and labeled the log sequences. (false positives/negatives) may be introduced during the manual labeling process. However, as the engineers are experienced professionals of the product team who maintain the service system, we believe the amount of noise is small (if it exists)

It also sets Log3C against the two baselines that come back later on this page. Furthermore, we compare our method with two typical methods: PCA \[41] and Invariants Mining \[23]. All these three methods are unsupervised, log-based problem identification methods. PCA projects the log sequence vectors into a subspace. If the projected vector is far from the majority, it is considered as a problem. Invariants Mining extracts the linear relations (invariants) between log event occurrences, which hypothesizes that log events are often pairwise generated. For example, when processing files, "File A is opened" and "File A is closed" should be printed as a pair. Log sequences that violate the invariants are regarded as problematic. Log3C achieves good recalls (similar to those achieved by two comparative methods) and surpasses the comparative methods concerning precision and F1-measure.

The same group built tools for the rest of the log lifecycle. [Logzip](https://github.com/logpai/logzip) is an optimized log compression tool via iterative clustering \[ASE'19], and its [paper](https://arxiv.org/abs/1910.00409) is Logzip: Extracting Hidden Structures via Iterative Clustering for Log Compression. Logzip is an (personal note seems to be offline) efficient compression tool specific for log files. It compresses log files by utilizing the inherent structures of raw log messages, and thereby achieves a high compression ratio. The results show that logzip can save about half of the storage space on average over traditional compression tools. Meanwhile, the design of logzip is highly parallel and only incurs negligible overhead. In addition, we share our industrial experience of applying logzip to Huawei's real products.

Logadvisor moves upstream, to where logs are written. [paper1](https://jiemingzhu.github.io/pub/qfu_icse2014.pdf) is Where Do Developers Log? An Empirical Study on Logging Practices in Industry, which argues that it is crucial to avoid logging too little or too much and that developers need informed decisions on where to log and what to log. [2](https://jiemingzhu.github.io/pub/jmzhu_icse2015.pdf) is Learning to Log: Helping Developers Make Informed Logging Decisions, from the CUHK Sub-Lab of the Ministry of Education Key Laboratory of High Confidence Software Technologies. Our goal, referred to as “learning to log”, is to automatically learn the common logging practice as a machine learning model, and then leverage the model to guide developers to make logging decisions during new development. The model is built in three steps:

1. Labels: logging method (e.g., Console.Writeline())
2. Features: we need to extract useful features (e.g., exception type) from the collected code snippets for making logging decisions,
3. Train / suggest

The two Logadvisor figures show that setup.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-e51ab9f4533ccf12f5ab89b2aaccee1f17803063%2Fgimg-1f9907addd4f.png?alt=media" alt=""><figcaption><p>Logadvisor.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-b0946193073481a83f30ee39c1212e124fdf8b38%2Fgimg-5ea6dadfb401.png?alt=media" alt=""><figcaption><p>Logadvisor.</p></figcaption></figure>

A related dataset covers the text of the log line itself. [Logging descriptions -](https://github.com/logpai/LoggingDescriptions) This repository maintains a set of `<code, log>` pairs extracted from popular open-source projects, which are amendable to logging description generation research.

The tool that ties parsing to detection is (REALLY GOOD) [Loglizer](https://github.com/logpai/loglizer). Its [paper](https://jiemingzhu.github.io/pub/slhe_issre2016.pdf) is Experience Report: System Log Analysis for Anomaly Detection, which starts from the fact that developers traditionally inspect logs manually with keyword search and rule matching, and that modern systems produce too many logs for that. The [git demo](https://github.com/logpai/loglizer/tree/master/demo) is the demo folder of the repo. Loglizer is a machine learning-based log analysis toolkit for automated anomaly detection.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-6f33bdb8360227b82df54cc651ed3b422c63ac73%2Fgimg-ef291d4c8190.png?alt=media" alt=""><figcaption><p>Loglizer.</p></figcaption></figure>

## LogParser benchmark

With so many parsers around, the question is which one to use. [LogParser](https://github.com/logpai/logparser) is a benchmark for log parsers using 13 models on 16 datasets. **Important insights:** Drain is fastest, most performing on most datasets (9/16). Fitting parameters should be adapted, which what makes drain the most performing. The benchmark also calls for more demanding metrics. The papers behind it:

* \[ICSE'19] Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, Michael R. Lyu. [Tools and Benchmarks for Automated Log Parsing](https://arxiv.org/pdf/1811.03509.pdf). International Conference on Software Engineering (ICSE), 2019.
* \[TDSC'18] Pinjia He, Jieming Zhu, Shilin He, Jian Li, Michael R. Lyu. [Towards Automated Log Parsing for Large-Scale Log Data Analysis](https://jiemingzhu.github.io/pub/pjhe_tdsc2017.pdf). IEEE Transactions on Dependable and Secure Computing (TDSC), 2018.
* \[ICWS'17] Pinjia He, Jieming Zhu, Zibin Zheng, Michael R. Lyu. [Drain: An Online Log Parsing Approach with Fixed Depth Tree](https://jiemingzhu.github.io/pub/pjhe_icws2017.pdf). IEEE International Conference on Web Services (ICWS), 2017.
* \[DSN'16] Pinjia He, Jieming Zhu, Shilin He, Jian Li, Michael R. Lyu. [An Evaluation Study on Log Parsing and Its Use in Log Mining](https://jiemingzhu.github.io/pub/pjhe_dsn2016.pdf). IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), 2016.

The benchmark figure summarizes the comparison.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-7a29179f0c9450b021f6725913320a3a0e0307f1%2Fgimg-29516eda17cc.png?alt=media" alt=""><figcaption><p>LogParser benchmark.</p></figcaption></figure>

Beyond the benchmarked parsers, deep models read the logs directly. [Gpt3 with logs](https://www.zebrium.com/blog/using-gpt-3-with-zebrium-for-plain-language-incident-root-cause-from-logs) was the Zebrium post on plain-language incident root cause from logs; the address now lands on Skylar Advisor, which combines real-time observability data, operational context, and customer-owned knowledge into explainable AI guidance for IT operations. [DeepLog](https://www.cs.utah.edu/~lifeifei/papers/deeplog.pdf) is DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning, which treats the system log, recorded at critical points to help debug failures and perform root cause analysis, as the input for anomaly detection. GitHub - wuyifan18/DeepLog is a Pytorch Implementation of DeepLog. ( [git](https://github.com/wuyifan18/DeepLog)

[Log2vec](https://netman.aiops.org/wp-content/uploads/2020/05/Log2Vec-icccn20.pdf) is the Tsinghua University paper with Federico Zaiter, Bingjin Chen, and Dan Pei, and its code ([git](https://github.com/NetManAIOps/Log2Vec)) is a distributed representation method for online logs.

Three earlier sources come back here as plain addresses. logpaI loglizer: An Evaluation Study on Log Parsing and Its Use in Log Mining is by PinjiaHe, whose profile is <https://github.com/PinjiaHe>. The toolkit is a machine learning toolkit for log-based anomaly detection \[ISSRE'16] - logpai/loglizer, **(REALLY GOOD)**, with the Loglizer paper linked above, at <https://github.com/logpai/loglizer>. The next section is based on 3 things we learned about applying word vectors to logs, archived at <https://web.archive.org/web/20180629032123/https://gab41.lab41.org/three-things-we-learned-about-applying-word-vectors-to-computer-logs-c199070f390b>

## Word vectors on logs

Instead of parsing templates first, the Lab41 write-up, 3 things we learned about applying word vectors to logs, runs GloVe on the log stream. GloVe consistently identified approximately 50 percent or more of the seeded events in the synthetic data as either exact or as valid sub-sequence matches. GloVe tended to nominate a limited number of template sequences that weren’t related to seeded events and many of those were tied to high frequency templates. When we tested GloVe against a generated data set with multiple SSH sessions in an auditd file, GloVe correctly proposed a single event that included all of the auditd record types defined in the SSH user login lifecycle.

The catch is that Glove produces sub sequences that needs to be stitched to create a match, as the figure shows.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-bfee430bf5b3ade552e870919298f2c74a062fa0%2Fgimg-eb48f204a54e.png?alt=media" alt=""><figcaption><p>Glove produces sub sequences that needs to be stitched to create a match.</p></figcaption></figure>

On speed, Glove is faster than paris and fp growth. On accuracy, their clustering method misclassified.

## Feature extraction windows

Whether events come from a parser or from word vectors, they have to be grouped into sequences before a model can use them. Feature extraction uses a fixed window, a sliding window, or a session window.

Fixed window: Both fixed windows and sliding windows are based on timestamp, which records the occurrence time of each log. Each fixed window has its size, which means the time span or time duration. As shown in Figure 1, the window size is Δt, which is a constant value, such as one hour or one day. Thus, the number of fixed windows depends on the predefined window size. Logs that happened in the same window are regarded as a log sequence.

Sliding window: Different from fixed windows, sliding windows consist of two attributes: window size and step size, e.g., hourly windows sliding every five minutes. In general, step size is smaller than window size, therefore causing the overlap of different windows. Figure 1 shows that the window size is ΔT , while the step size is the forwarding distance. The number of sliding windows, which is often larger than fixed windows, mainly depends on both window size and step size. Logs that occurred in the same sliding window are also grouped as a log sequence, though logs may duplicate in multiple sliding windows due to the overlap.

Session window: Compared with the above two windowing types, session windows are based on identifiers instead of the timestamp. Identifiers are utilized to mark different execution paths in some log data. For instance, HDFS logs with block\_id record the allocation, writing, replication, deletion of certain block. Thus, we can group logs according to the identifiers, where each session window has a unique identifier

## PCA for log anomaly detection

Once logs are windowed into event count vectors, the PCA baseline that Log3C compared against can be applied. The same notes are in [Anomaly Detection](/predictive-ml/anomaly-detection) and [PCA](/predictive-ml/dimensionality-reduction-methods#pca).

There are many Supervised methods and most importantly a cool unsupervised method - > PCA for anomaly based on the length of the projected transformed sample vector by dividing the first and last PC vectors. PCA was first applied in log-based anomaly detection by Xu et al. \[47]. In their anomaly detection method, each log sequence is vectorized as an event count vector. After that, PCA is employed to find patterns between the dimensions of event count vectors. Employing PCA, two subspace are generated, namely normal space Sn and anomaly space Sa. Sn is constructed by the first k principal components and Sn is constructed by the remaining (n−k), where n is the original dimension. Then, the projection ya = (1−P P T )y of an event count vector y to Sa is calculated, where P = \[v1,v2, ...,vk,] is the first k principal components. If the length of ya is larger

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* 3 things we learned about applying word vectors to logs. This address no longer opens: <https://gab41.lab41.org/three-things-we-learned-about-applying-word-vectors-to-computer-logs-c199070f390b#.bk8wnk7pr>
* logpaI loglizer: An Evaluation Study on Log Parsing and Its Use in Log Mining. This address no longer opens: <https://github.com/PinjiaHe>. This address no longer opens: <https://pinjiahe.github.io/papers/DSN16.pdf>
* **Logadvisor -** 2. This address no longer opens: <http://jmzhu.logpai.com/pub/jmzhu\\_icse2015.pdf>
* **(REALLY GOOD)** Loglizer paper. This address no longer opens: <https://github.com/logpai/loglizer>. This address no longer opens: <http://jmzhu.logpai.com/pub/slhe\\_issre2016.pdf>


# Forecasting

A time series is a sequence of measurements where the order matters, so predicting the next value means first understanding what the series is made of. The page starts with the components of a series, transforms, stationarity, splitting, and the windows used to smooth and weight it, then decomposition and short series. It then moves to forecasting methods and how to evaluate them, classification with DTW, Kalman filters, LSTMs, the tools, and finally Pandas manipulation.

The same notes are in [Anomaly Detection](/predictive-ml/anomaly-detection), [Time series entropy](/data/analytics-intro/information-theory#time-series-entropy), and [Timeseries](#timeseries).

The simplest baseline is a series that just wanders. MachineLearningMastery's A Gentle Introduction to the Random Walk for Times Series Forecasting with Python is the [Random walk](https://machinelearningmastery.com/gentle-introduction-random-walk-times-series-forecasting-python/) reference, and the figure below illustrates it.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-d35be53403efbac803b0ee23db7bf80ad2a9ab0b%2Fgimg-71b2444e4eb4.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

Anything beyond a random walk has structure that can be pulled apart. The [Time series decomposition book](https://otexts.com/fpp2/forecasting-decomposition.html) is section 6.8, Forecasting with decomposition, of Forecasting: Principles and Practice (2nd ed). [Mastery on ts decomposition](https://machinelearningmastery.com/decompose-time-series-data-trend-seasonality/) is MachineLearningMastery's How to Decompose Time Series Data into Trend and Seasonality.

### [Time Series Components](http://machinelearningmastery.com/time-series-forecasting/)

Decomposition needs names for the parts. The heading link is Jason Brownlee's What Is Time Series Forecasting?, which argues that time series forecasting is an important and often neglected area of machine learning, neglected because the time component makes the problems harder to handle. It names four components:

1. Level. The baseline value for the series if it were a straight line.
2. Trend. The optional and often linear increasing or decreasing behavior of the series over time.
3. Seasonality. The optional repeating patterns or cycles of behavior over time.
4. Noise. The optional variability in the observations that cannot be explained by the model.

All time series have a level, most have noise, and the trend and seasonality are optional.

With the components named, a series can be framed as supervised learning. Take a one step forecast using a window of “1” and a typical sample “time, measure1, measure2”. Linear/nonlinear classifiers predict a single output value, using the t-1 previous line, i.e., “measure1 t, measure 2 t, measure 1 t+1, measure 2 t+1 (as the class)”. Neural networks predict multiple output values, i.e., “measure1 t, measure 2 t, measure 1 t+1(class1), measure 2 t+1(class2)”.

One-Step Forecast: This is where the next time step (t+1) is predicted.

Multi-Step Forecast: This is where two or more future time steps are to be predicted.

A multi-step forecast using a window of “1” and a typical sample “time, measure1” uses the current value input and labels it with the two future input labels: “measure1 t, measure1 t+1(class), measure1 t+2(class1)”.

This article explains about ML Methods for Sequential Supervised Learning - Six methods that have been applied to solve sequential supervised learning problems:

The same notes are in [CONDITIONAL RANDOM FIELDS (CRF)](/predictive-ml/probabilistic-models#conditional-random-fields-crf) and [MARKOV MODELS / HIDDEN MARKOV MODEL](/predictive-ml/probabilistic-models#markov-models--hidden-markov-model).

1. sliding-window methods - converts a sequential supervised problem into a classical supervised problem
2. recurrent sliding windows
3. hidden Markov models
4. maximum entropy Markov models
5. input-output Markov models
6. conditional random fields
7. graph transformer networks

The article is the Oregon State University paper at <http://web.engr.oregonstate.edu/~tgd/publications/mlsd-ssspr.pdf>, which formalizes the principal learning tasks for sequential data and describes the methods developed for them.

##

### [Data Transformations](https://www.otexts.org/fpp/2/4)

Before any of those methods, the raw series is often transformed. The forecasting textbook section in the heading covers the Log and Box cox transforms, the Back transform that returns forecasts to the original scale, Calendrical adjustments, and Inflation adjustment.

Transforming time series data to tabular (in order to use tabular based approach) is covered at <https://towardsdatascience.com/approaching-time-series-with-a-tree-based-model-87c6d1fb6603>.

### STATIONARY TIME SERIES

Most classical models assume the transformed series is stationary. [What is?](https://machinelearningmastery.com/time-series-data-stationary-python/) A time series without a trend or seasonality, in other words non-stationary has a trend or seasonality

There are ways to [remove the trend and seasonality](https://machinelearningmastery.com/difference-time-series-dataset-python/), i.e., take the difference between time points. The steps are:

1. T+1 - T
2. Bigger lag to support seasonal changes
3. pandas.diff()
4. Plot a histogram, plot a log(X) as well.
5. Test for the unit root null hypothesis - i.e., use the Augmented dickey fuller test to determine if two samples originate in a stationary or a non-stationary (seasonal/trend) time series

Shay on stationary time series, AR, ARMA used to be linked here; the address no longer opens and is kept at the end of the page.

For a decomposition that handles seasonality robustly, (amazing) [STL](https://otexts.com/fpp2/stl.html) and more.

### SPLITTING TIME SERIES DATA

A stationary series still cannot be split at random, because the future must not leak into training. SK-lego With a gap is the splitter for that; its old page is kept at the end. It is now with even timeseries split by group, as Tomer Gabay's A highly anticipated Time Series Cross-validator is finally here explains: unevenly spread Time Series data is no longer a problem for cross-validation. <https://towardsdatascience.com/a-highly-anticipated-time-series-cross-validator-is-finally-here-7dc99f672736>

### [Rolling window analysis](https://link.springer.com/chapter/10.1007%2F978-0-387-32348-0_9)

Splitting asks whether a model generalizes forward in time; a rolling window asks whether its parameters stay the same over time. The chapter in the heading puts it this way:

“compute parameter estimates over a rolling window of a fixed size through the sample. If the parameters are truly constant over the entire sample, then the estimates over the rolling windows should not be too different. If the parameters change at some point during the sample, then the rolling estimates should capture this instability”

### [Moving average window](https://www.otexts.org/fpp/6/2)

A window can also smooth the series itself. The moving average is used to estimate the trend cycle. The window size is a choice, 3-5-7-9? If its too large its going to flatten the curve, too low its going to be similar to the actual curve. There is also the two tier moving average, first 4 then 2 on the resulted moving average.

[Visual example](https://www.youtube.com/watch?v=_YXoRTQQI3U) of ARIMA algorithm - captures the time series trend or forecast.

### Weighted “window”

A plain moving average weights every point in the window equally; a weighted window lets older points count less. The tool here is 1, scikit-lego with a decay estimator, whose page is kept at the end. The figure shows the decay.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-e4bc90dce81fcb169d56fa7c1c615d1132b398cd%2Fgimg-30e14a2426a2.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

### Decomposition

Smoothing extracts the trend; seasonality needs its own basis. Creating curves to explain a complex seasonal fit is the scikit-lego repeating basis function approach, whose page is kept at the end. The two figures show the curves and the fit.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-542dc7542cd3f5ab1991bc78105ee22c49cd04db%2Fgimg-2e883a2cef54.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-87a0f92ff2a8c6199ba5479305810b7636907724%2Fgimg-3c78cf25bcb4.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

### SHORT TIME SERIES

All of the above assumes enough history, and many real series are short. [Short time series](https://robjhyndman.com/hyndsight/short-time-series/) is Rob J Hyndman's Fitting models to short time series. PDarima - Pmdarima‘s auto\_arima function is extremely useful when building an ARIMA model as it helps us identify the most optimal p,d,q parameters and return a fitted ARIMA model; Muriel Kosaka's Efficient Time-Series Using Python's Pmdarima Library demonstrates the efficiency of pmdarima's auto\_arima() function compared to implementing a traditional ARIMA model: <https://towardsdatascience.com/efficient-time-series-using-pythons-pmdarima-library-f6825407b7f0>. How little data is enough is the question of [Min sample size for short seasonal time series](https://robjhyndman.com/papers/shortseasonal.pdf).

[More mastery on short time series.](https://machinelearningmastery.com/time-series-forecasting-methods-in-python-cheat-sheet/) is MachineLearningMastery's 11 Classical Time Series Forecasting Methods in Python (Cheat Sheet), which lists:

1. Autoregression (AR)
2. Moving Average (MA)
3. Autoregressive Moving Average (ARMA)
4. Autoregressive Integrated Moving Average (ARIMA)
5. Seasonal Autoregressive Integrated Moving-Average (SARIMA)
6. Seasonal Autoregressive Integrated Moving-Average with Exogenous Regressors (SARIMAX)
7. Vector Autoregression (VAR)
8. Vector Autoregression Moving-Average (VARMA)
9. Vector Autoregression Moving-Average with Exogenous Regressors (VARMAX)
10. Simple Exponential Smoothing (SES)
11. Holt Winter’s Exponential Smoothing (HWES)

Auto ARIMA also scales to many series: Michael Keith's Multiple Series? Forecast Them together with any Sklearn Model shows how to use Python to forecast the trends of multiple series at the same time, 1. <https://towardsdatascience.com/time-series-forecasting-using-auto-arima-in-python-bb83e49210cd>

Predicting actual Values of time series using observations is a different angle: [Using kalman filters](https://www.youtube.com/watch?v=CaCcOwJPytQ) - explains the concept etc, 1 out of 55 videos.

### [Forecasting methods](https://www.otexts.org/fpp/2/3)

Before reaching for ARIMA, the textbook sets simple benchmarks that any model must beat:

* Average: Forecasts of all future values are equal to the mean of the historical data.
* Naive: Forecasts are simply set to be the value of the last observation.
* Seasonal Naive: forecast to be equal to the last observed value from the same season of the year
* Drift: A variation on the naïve method is to allow the forecasts to increase or decrease over time, the drift is set to be the average change seen in the historical data.

### [Evaluate forecast accuracy](https://www.otexts.org/fpp/2/5)

A benchmark only matters once there is a way to score it. The figure summarizes the accuracy measures from the textbook section in the heading.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-ba1199f1ddd2dda127c976fa2c9a540a5c5ad287%2Fgimg-4520aee72acc.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

Regression forecasts usually carry calendar effects as dummy variables, and that is where beginners trip. Dummy variables: sunday, monday, tues,wed,thurs, friday. NO SATURDAY! Notice that only six dummy variables are needed to code seven categories. That is because the seventh category (in this case Sunday) is specified when the dummy variables are all set to zero. Many beginners will try to add a seventh dummy variable for the seventh category. This is known as the "dummy variable trap" because it will cause the regression to fail.

The same trick handles other effects:

* Outliers: If there is an outlier in the data, rather than omit it, you can use a dummy variable to remove its effect. In this case, the dummy variable takes value one for that observation and zero everywhere else.
* Public holidays: For daily data, the effect of public holidays can be accounted for by including a dummy variable predictor taking value one on public holidays and zero elsewhere.
* Easter: is different from most holidays because it is not held on the same date each year and the effect can last for several days. In this case, a dummy variable can be used with value one where any part of the holiday falls in the particular time period and zero otherwise.
* Trading days: The number of trading days in a month can vary considerably and can have a substantial effect on sales data. To allow for this, the number of trading days in each month can be included as a predictor. An alternative that allows for the effects of different days of the week has the following predictors. # Mondays in month;# Tuesdays in month;# Sundays in month.
* Advertising: $advertising for previous month;$advertising for two months previously

### CLASSIFICATION

Forecasting predicts the next value; classification labels a whole series, and that needs a distance between series. [Stackexchange](https://stats.stackexchange.com/questions/131281/dynamic-time-warping-clustering/131284) - Yes, you can use DTW approach for classification and clustering of time series. I've compiled the following resources, which are focused on this very topic (I've recently answered a similar question, but not on this site, so I'm copying the contents here for everybody's convenience):

* UCR Time Series Classification/Clustering: main page, software page and corresponding paper
* Time Series Classification and Clustering with Python: a blog post
* Capital Bikeshare: Time Series Clustering: another blog post
* Jupyter Notebook Viewer. Time Series Classification and Clustering: [ipython notebook](http://nbviewer.ipython.org/github/alexminnaar/time-series-classification-and-clustering/blob/master/Time%20Series%20Classification%20and%20Clustering.ipynb)
* Dynamic Time Warping using rpy and Python:. [another blog post](https://nipunbatra.wordpress.com/2013/06/09/dynamic-time-warping-using-rpy-and-python)
* Mining Time-series with Trillions of Points: Dynamic Time Warping at Scale: another blog post
* Time Series Analysis and Mining in R (to add R to the mix): yet another blog post
* And, finally, two tools implementing/supporting DTW, to top it off: R package and Python module

Two of those answers have addresses of their own. The Mining Time-series with Trillions of Points: Dynamic Time Warping at scale post, another blog post, takes a similarity measure that's already well-known and runs it at scale: <http://practicalquant.blogspot.com/2012/10/mining-time-series-with-trillions-of.html>. Time Series Analysis and Mining with R on blog.RDataMining.com is yet another blog post: <http://rdatamining.wordpress.com/2011/08/23/time-series-analysis-and-mining-with-r>. The Python module from that list is kept at the end.

Tree models are the other way to classify or regress on series. Satya Pattnaik's A LightGBM Autoregressor — Using Sktime is at <https://towardsdatascience.com/a-lightgbm-autoregressor-using-sktime-6402726e0e7b>. To give those models a better notion of time, Marco Cerliani's Time2vec for Time Series features encoding shows how to learn a valuable representation of time for your Machine Learning Model: <https://towardsdatascience.com/time2vec-for-time-series-features-encoding-a03a4f3f937e>

### [Kalman filters in matlab](https://www.youtube.com/watch?v=4OerJmPpkRg)

The Kalman filter video from the short-series section has a MATLAB counterpart. The heading link is MATLAB's State Observers | Understanding Kalman Filters, Part 2.

### [LTSM for time series](http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/)

Kalman filters model hidden state explicitly; an LSTM learns it. The heading link is Jason Brownlee's Time Series Prediction with LSTM Recurrent Neural Networks in Python with Keras, which notes that time series add the complexity of a sequence dependence among the input variables, and that the LSTM is a recurrent network designed to handle it.

The same notes are in [Deep Neural Time Series](/deep-learning/deep-neural-time-series) and [LSTM](/deep-learning/recurrent-nets#lstm).

There are three types of gates within a unit:

* Forget Gate: conditionally decides what information to throw away from the block.
* Input Gate: conditionally decides which values from the input to update the memory state.
* Output Gate: conditionally decides what to output based on input and the memory of the block.

Using lstm to predict sun spots, has some autocorrelation usage. [Part 1](https://www.business-science.io/timeseries-analysis/2018/04/18/keras-lstm-sunspots-time-series-prediction.html) teaches time series analysis with Keras LSTM deep learning by predicting sunspots ten years into the future. [Part 2](https://www.business-science.io/timeseries-analysis/2018/07/01/keras-lstm-sunspots-part2.html) continues the KERAS LSTM deep learning time series analysis on the NASA sunspots data set.

### TOOLS

Every method above has a library. SKtime - is a sk-based api, with an introduction on medium (kept at the end), and it integrates algos from tsfresh and tslearn. (really good) A LightGBM Autoregressor — Using Sktime, explains about the basics in time series prediction, splitting, next step, delayed step, multi step, deseason. [SKtime-DL - using keras and DL](https://github.com/sktime/sktime-dl) is marked DEPRECATED, now in sktime - it was the companion package for deep learning based on TensorFlow.

For features and distances there are two older libraries. [TSFresh](http://tsfresh.readthedocs.io) - extracts 1200 features, filters them using FDR for time series classification etc. [TSlearn](http://tslearn.readthedocs.io) - DTW, shapes, shapelets (keras layer), time series kmeans/clustering/svm/svr/KNN/bary centers/PAA/SAX

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-e5bab4ade3575e427cd0a8f5dd355e2dbfba98d5%2Fgimg-0700069a56bf.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

The DTW from the classification section has a dedicated library. [DTAIDistance](https://dtaidistance.readthedocs.io/en/latest/index.html) - Library for time series distances (e.g. Dynamic Time Warping) used in the [DTAI Research Group](https://dtai.cs.kuleuven.be/). The library offers a pure Python implementation and a faster implementation in C. The C implementation has only Cython as a dependency. It is compatible with Numpy and Pandas and implemented to avoid unnecessary data copy operations. Its clustering modules are [dtaidistance.clustering.hierarchical](https://dtaidistance.readthedocs.io/en/latest/modules/clustering/hierarchical.html), [Ddtaidistance.clustering.kmeans](https://dtaidistance.readthedocs.io/en/latest/modules/clustering/kmeans.html), and [Dtaidistance.clustering.medoids](https://dtaidistance.readthedocs.io/en/latest/modules/clustering/medoids.html).

For forecasting in one package, [Darts](https://unit8co.github.io/darts/) is a Python library for user-friendly forecasting and anomaly detection on time series. See its [Forecasting models](https://unit8co.github.io/darts/#forecasting-models) & [Examples](https://unit8co.github.io/darts/#example-usage).

The same distances can also identify anomalies, outliers or abnormal behaviour (see for example the [anomatools package](https://github.com/Vincent-Vercruyssen/anomatools), a toolbox for anomaly detection).

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-0e996b6a54ffe4334c139edac838c61f94b014c4%2Fgimg-bf36052b3a66.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

When a few labels are available, there is Semi supervised with DTAIDistance - Active semi-supervised clustering. The recommended method for perform active semi-supervised clustering using DTAIDistance is to use the COBRAS for time series clustering: <https://github.com/ML-KULeuven/cobras>. COBRAS is a library for semi-supervised time series clustering using pairwise constraints, which natively supports both dtaidistance.dtw and kshape.

Warping can also be learned. [Affine warp](https://github.com/ahwillia/affinewarp), a neural net with time warping - as part of the following manuscript, which focuses on analysis of large-scale neural recordings (though this code can be also be applied to many other data types). [Neural warp](https://github.com/josifgrabocka/neuralwarp) is a supporting website for the paper "NeuralWarp: Time-Series Similarity with Warping Networks", and the paper itself is [NeuralWarp](https://arxiv.org/pdf/1812.08306.pdf) Time Series Similarity with Warping Networks.

The tools all rest on one idea about features. [A great introduction into time series](https://medium.com/making-sense-of-data/time-series-next-value-prediction-using-regression-over-a-rolling-window-228f0acae363) - “The approach is to come up with a list of features that captures the temporal aspects so that the auto correlation information is not lost.” basically tells us to take sequence features and create (auto)-correlated new variables using a time window, i.e., “Time series forecasts as regression that factor in autocorrelation as well.”. we can transform raw features into other type of features that explain the relationship in time between features. we measure success using loss functions, MAE RMSE MAPE RMSEP AC-ERROR-RATE

[Interesting idea](http://blog.kaggle.com/2016/02/03/rossmann-store-sales-winners-interview-2nd-place-nima-shahbazi/) on how to define ‘time series’ dummy variables that utilize beginning\end of certain holiday events, including important information on what NOT to filter even if it seems insignificant, such as zero sales that may indicate some relationship to many sales the following day.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-d65aef14cd59ff71e9d00e920301ada803caeef8%2Fgimg-9a9a0f71a786.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

Those features only make sense against the patterns they are meant to capture. [Time series patterns: ](https://www.otexts.org/fpp/2/1)A trend (a,b,c) exists when there is a long-term increase or decrease in the data. A seasonal (a - big waves) pattern occurs when a time series is affected by seasonal factors such as the time of the year or the day of the week. The monthly sales induced by the change in cost at the end of the calendar year. A cycle (a) occurs when the data exhibit rises and falls that are not of a fixed period - sometimes years.

[Some statistical measures](https://www.otexts.org/fpp/2/2) (mean, median, percentiles, iqr, std dev, bivariate statistics - correlation between variables)

Bivariate Formula: this correlation measures the extent of a linear relationship between two variables. high number = high correlation between two variable. The value of r always lies between -1 and 1 with negative values indicating a negative relationship and positive values indicating a positive relationship. Negative = decreasing, positive = increasing.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-6d0f9424141d20427f111370ec789998034c558b%2Fgimg-49e1a9c9f846.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

But correlation can LIE, the following has 0.8 correlation for all of the graphs:

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-3cb6f908b09857b20ab62c2874a0b374992355a9%2Fgimg-6ef2dbb63a24.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

Autocorrelation measures the linear relationship between lagged values of a time series. L8 is correlated, and has a high measure of 0.83. White-noise has autocorrelation of 0.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-a305b531f3e63b4ee7228a76f70686bd466136b6%2Fgimg-0ff80aebaef9.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

## Timeseries

All of the above starts with a series loaded and shaped in Pandas. The same notes are in [Timeseries](/predictive-ml/forecasting).

(good) [Pandas time series manipulation](https://medium.com/data-science/practical-guide-for-time-series-analysis-with-pandas-196b8b46858f) is the Practical Guide for Time Series Analysis with Pandas. [Using resample](https://medium.com/data-science/using-the-pandas-resample-function-a231144194c4) is Jeremy Chow's introductory dive into the technical aspects of the pandas resample function for datetime manipulation, meant as readable pseudo-documentation for those who would rather not dig through the pandas source code. The figure is his.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-94a7dd3aa31ed75c186f9a3b9781c3514baccc06%2Fgimg-f2b33a13212b.png?alt=media" alt=""><figcaption><p>Pandas resample illustration.</p></figcaption></figure>

Past basic TS manipulation, the next step is to [Fill missing ts gaps, or how to resample](https://stackoverflow.com/questions/32241692/fill-missing-timeseries-data-using-pandas-or-numpy), the Stack Overflow question on filling missing timeseries data using pandas or numpy.

From Pandas the series moves into SCI-KIT LEARN. Pipeline t [o json 1](https://cmry.github.io/notes/serialize) is Scikit-learn Pipeline Persistence and JSON Serialization, which opens by thanking Sebastian Raschka and Chris Wagner for the text and code behind it, and [2](https://cmry.github.io/notes/serialize-sk) is Part II, a follow-up to that post. [cuML](https://github.com/rapidsai/cuml) - Multi gpu, multi node-gpu alternative for SKLEARN algorithms, and the [Gpu TSNE ^](https://www.reddit.com/r/MachineLearning/comments/e0j9cb/p_2000x_faster_rapids_tsne_3_hours_down_to_5/) thread goes with it. [Awesome code examples](http://machinelearningmastery.com/get-your-hands-dirty-with-scikit-learn-now/) about using svm/knn/naive/log regression in sklearn in python, i.e., “fitting a model onto the data”.

When one machine is not enough, dask handles parallel computing with task scheduling. [Parallelism of numpy, pandas and sklearn using dask and clusters](https://github.com/dask/dask) is the repo, the [Webpage](https://dask.pydata.org/en/latest/) is the Dask documentation, the dask-ml [docs](http://dask-ml.readthedocs.io/en/latest/index.html) now report that the API page has moved, and there is an [example in jupyter](https://hub.mybinder.org/user/dask-dask-examples-6bi4j3qj/notebooks/machine-learning.ipynb).

Also Insanely fast, [see here](https://www.youtube.com/watch?v=5Zf6DQaf7jk), Tom Augspurger's talk on distributed scikit learn.

Finally, the pipelines themselves can be written in a functional api for sk learn, using pipelines. thank you sk-lego.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-8e738d87ce916f94bd6760ffa29204206fe6df6d%2Fgimg-83b1fcbbc61e.png?alt=media" alt=""><figcaption><p>SK-Lego pipeline illustration.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-45359f3fa58fec81532a1d465bda47aad9a2d183%2Fgimg-9538a8273118.png?alt=media" alt=""><figcaption><p>SK-Lego pipeline illustration.</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-91a17224965440533da968ef50fb2d715ae91e74%2Fgimg-b59ff3ac85bb.png?alt=media" alt=""><figcaption><p>SK-Lego pipeline illustration.</p></figcaption></figure>

Images by [SK-Lego](https://medium.com/@jeremyrchow)

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

\- medium. This address no longer opens: <https://towardsdatascience.com/sktime-a-unified-python-library-for-time-series-machine-learning-3c103c139a55> - With a gap. This address no longer opens: <https://scikit-lego.readthedocs.io/en/latest/timegapsplit.html> - Creating. This address no longer opens: <https://scikit-lego.readthedocs.io/en/latest/preprocessing.html#Repeating-Basis-Function-Transformer> - 1, scikit-lego with a decay estimator. This address no longer opens: <https://scikit-lego.readthedocs.io/en/latest/meta.html#Decayed-Estimation> - Shay on stationary time series, AR, ARMA. This address no longer opens: <https://towardsdatascience.com/stationarity-in-time-series-analysis-90c94f27322> - Python module. This address no longer opens: <http://mlpy.sourceforge.net/>


# Time Series Search

Searching, grouping, and monitoring time series all depend on one question: how far apart are two series? The page starts with Dynamic Time Warping (DTW), the distance that answers it, along with its myths and code, then uses that distance for clustering series, and ends with anomaly detection on time series and the toolkits for it.

### Dynamic Time Warping (DTW)

Euclidean distance compares two series point by point and breaks as soon as one is shifted or stretched, so the first tool is a better distance. The same notes are in [Clustering Algorithms](/predictive-ml/clustering-algorithms) and [Distance](/data/processing-intro/feature-engineering#distance).

DTW, ie., how to compute a better distance for two time series. Before using it, it is worth knowing the three myths of using DTW, from the paper <http://alumni.cs.ucr.edu/~ratana/RatanamC.pdf>. The paper recalls that DTW has long been known in the speech recognition community, where it allows a non-linear mapping of one signal to another by minimizing the distance between the two, and that it was introduced into data mining for classification, clustering, and anomaly detection on time series. The three myths:

Myth 1: The ability of DTW to handle sequences of different lengths is a great advantage, and therefore the simple lower bound that requires different-length sequences to be reinterpolated to equal length is of limited utility \[10]\[19]\[21]. In fact, as we will show, there is no evidence in the literature to suggest this, and extensive empirical evidence presented here suggests that comparing sequences of different lengths and reinterpolating them to equal length produce no statistically significant difference in accuracy or precision/recall.

Myth 2: Constraining the warping paths is a necessary evil that we inherited from the speech processing community to make DTW tractable, and that we should find ways to speed up DTW with no (or larger) constraints\[19]. In fact, the opposite is true. As we will show, the 10% constraint on warping inherited blindly from the speech processing community is actually too large for real world data mining.

Myth 3: There is a need (and room) for improvements in the speed of DTW for data mining applications. In fact, as we will show here, if we use a simple lower bounding technique, DTW is essentially O(n) for data mining applications. At least for CPU time, we are almost certainly at the asymptotic limit for speeding up DTW.

The video below walks through DTW.

{% embed url="<https://www.youtube.com/watch?v=_K1OsqCicBY>" %}

To run it, the [Python code](https://github.com/alexminnaar/time-series-classification-and-clustering) repo holds time series classification and clustering code written in Python, with a [good tutorial.](http://nbviewer.ipython.org/github/alexminnaar/time-series-classification-and-clustering/blob/master/Time%20Series%20Classification%20and%20Clustering.ipynb) in the notebook viewer. For another function for dtw distance in python, Chathurangi Shyalika's [Medium](https://medium.com/datadriveninvestor/dynamic-time-warping-dtw-d51d1a1e4afc) post on Dynamic Time Warping (DTW) starts from what a time series is, a series of data points indexed in time order, and mentions prunedDTW sparseDTW and fastDTW. Library versions are [DTW in TSLEARN](https://tslearn.readthedocs.io/en/latest/user_guide/dtw.html#soft-dtw), the tslearn user guide chapter on Dynamic Time Warping that opens at soft-DTW, and [DynamicTimeWarping](https://dynamictimewarping.github.io/py-api/html/api/dtw.dtw.html#dtw.dtw), the dtw function in the dtw-python package documentation. The figure shows a warping alignment.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-dc858d80cb1f3cfe526ef65d6a28f4a6709a9822%2Fgimg-05a66ce85e85.png?alt=media" alt=""><figcaption><p>Figure.</p></figcaption></figure>

Once the distance exists, it can drive classification and clustering. The following answer is a (duplicate above in classification) of the forecasting page: [Stackexchange](https://stats.stackexchange.com/questions/131281/dynamic-time-warping-clustering/131284) - Yes, you can use DTW approach for classification and clustering of time series. I've compiled the following resources, which are focused on this very topic (I've recently answered a similar question, but not on this site, so I'm copying the contents here for everybody's convenience):

* UCR Time Series Classification/Clustering: main page, software page and corresponding paper
* Time Series Classification and Clustering with Python: a blog post
* Capital Bikeshare: Time Series Clustering: another blog post
* Jupyter Notebook Viewer. Time Series Classification and Clustering: [ipython notebook](http://nbviewer.ipython.org/github/alexminnaar/time-series-classification-and-clustering/blob/master/Time%20Series%20Classification%20and%20Clustering.ipynb)
* Dynamic Time Warping using rpy and Python:. [another blog post](https://nipunbatra.wordpress.com/2013/06/09/dynamic-time-warping-using-rpy-and-python)
* Mining Time-series with Trillions of Points: Dynamic Time Warping at Scale: another blog post
* Time Series Analysis and Mining in R (to add R to the mix): yet another blog post
* And, finally, two tools implementing/supporting DTW, to top it off: R package and Python module

Several of those answers have their own addresses. The main page is Welcome to the UCR Time Series Classification/Clustering Page, <http://www.cs.ucr.edu/~eamonn/time_series_data>. The software page is the UCR Suite for Time Series Subsequence Search, <http://www.cs.ucr.edu/~eamonn/UCRsuite.html>. The corresponding paper is Searching and Mining Trillions of Time Series Subsequences under Dynamic Time Warping, <http://www.cs.ucr.edu/~eamonn/SIGKDD_trillion.pdf>. Time Series Classification and Clustering with Python –, a blog post, is at <http://alexminnaar.com/2014/04/16/Time-Series-Classification-and-Clustering-with-Python.html>. Capital Bikeshare: Time Series Clustering, another blog post, shows that weekdays on the bike share network are very different from weekends: <http://ofdataandscience.blogspot.com/2013/03/capital-bikeshare-time-series-clustering.html>.

Two more clustering walkthroughs follow. Time Series Hierarchical Clustering using Dynamic Time Warping in Python - notebook, is kept at the end of the page along with its notebook. K-Means with DTW, probably fixed length vectors, using tslearn is at <https://towardsdatascience.com/how-to-apply-k-means-clustering-to-time-series-data-28d04a8f7da3>. (nice) [With time series](https://medium.com/@shachiakyaagba_41915/dynamic-time-warping-with-time-series-1f5c05fb8950) is Shachia Kyaagba's Dynamic Time Warping with Time Series, which starts from everyday series such as temperatures, commodity prices, and school grades.

### CLUSTERING TS

Those walkthroughs cluster whole series; clustering pieces of one series is where it goes wrong. The same notes are in [Clustering Algorithms](/predictive-ml/clustering-algorithms).

Clustering time series, subsequences with a rolling window, the pitfall., is the warning; that article is kept at the end of the page. For whole series, [Clustering using tslearn](https://tslearn.readthedocs.io/en/stable/user_guide/clustering.html) is the Time Series Clustering chapter of the tslearn documentation. When series differ in length, [Kmeans for variable length](https://medium.com/@iliazaitsev/how-to-classify-a-dataset-with-observations-of-various-length-96fab8e95baf) is How to Use K-means to Classify a Dataset with Non-Fixed Number of Features, built on the Wrist-Worn Accelerometer Dataset of 839 observations in 14 action classes such as Walking, Drinking, and Eating. Its code is the [notebook](https://github.com/devforfu/Blog/blob/master/trees/scikit_learn.py) in the i-zaitsev/Blog repo of projects supporting blog posts.

### ANOMALY DETECTION TS

With distances and clusters in hand, the last job is spotting the points or windows that do not fit. The same notes are in [Anomaly Detection](/predictive-ml/anomaly-detection).

Most methods assume stationarity first, and [What is stationary (process](https://en.wikipedia.org/wiki/Stationary_process) is the Wikipedia article on a stationary process. The forecasting baselines are in [mastery on arimas](https://machinelearningmastery.com/time-series-forecasting-methods-in-python-cheat-sheet/), MachineLearningMastery's 11 Classical Time Series Forecasting Methods in Python (Cheat Sheet). TS anomaly algos (stl, trees, arima) used to be linked here and is kept at the end.

A three-part series introduces the techniques. [AD techniques](https://medium.com/dp6-us-blog/anomaly-detection-techniques-c3817e8e7b2f) is the first part, part [2](https://medium.com/dp6-us-blog/anomaly-detection-techniques-part-ii-9a08b6562619) covers the types of anomalies found in digital marketing and an improvement on the earlier technique, the modified Z-score, and part [3](https://medium.com/dp6-us-blog/anomaly-detection-techniques-part-iii-d27e7b0d6c8a) asks how to apply those two techniques in a daily routine. [Z-score, modified z-score and iqr an intro why z-score is not robust](http://colingorrie.github.io/outlier-detection.html) is Colin Gorrie's Three ways to detect outliers, a collection of functions meant to make outlier detection in exploratory analysis accessible to non-specialists.

For a toolkit, [Adtk](https://adtk.readthedocs.io/en/stable/userguide.html) a sklearn-like toolkit with an amazing intro, various algorithms for non seasonal and seasonal, transformers, ensembles. [Awesome TS anomaly detection](https://github.com/rob-med/awesome-TS-anomaly-detection) is a list of tools & datasets for anomaly detection on time-series data. [Transfer learning toolkit](https://github.com/FuzhenZhuang/Transfer-Learning-Toolkit) is the Transfer Learning Toolkit for Primary Researchers, and its [paper and benchmarks](https://arxiv.org/pdf/1911.08967.pdf) is Transfer Learning Toolkit: Primers and Benchmarks.

A simpler approach fits a model while ignoring outliers. [Ransac is a good baseline](https://medium.com/@iamhatesz/random-sample-consensus-bd2bb7b1be75) random sample consensus for outlier detection; Tomasz Wrona's Random sample consensus motivates it with readings from a weather station that you want to fit a model to. [Ransac](https://medium.com/@angel.manzur/got-outliers-ransac-them-f12b6b5f606e) is another introduction, and [2](https://medium.com/@saurabh.dasgupta1/outlier-detection-using-the-ransac-algorithm-de52670adb4a) is Outlier detection using the RANSAC algorithm, which describes RANSAC as an iterative, non-deterministic algorithm for eliminating outliers that is commonly used in computer vision. Items 3 and 4 of that run are kept at the end. You can feed ransac with tsfresh/tslearn features.

[Anomaly detection for time series](https://medium.com/@jetnew/anomaly-detection-of-time-series-data-e0cb6b382e33) is Jet New's Anomaly Detection of Time Series Data, which opens with the Wikipedia definition: anomaly detection (also outlier detection) is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data. AD for TS, recommended by DTAIDistance, is [anomatools](https://github.com/Vincent-Vercruyssen/anomatools), a toolbox for anomaly detection.

STL decomposition is the next family. [AD where anomalies coincide with seasonal peaks!!](https://medium.com/@richa.mishr01/anomaly-detection-in-seasonal-time-series-where-anomalies-coincide-with-seasonal-peaks-9859a6a6b8ba) handles the hard case in seasonal time series. [AD challenges, stationary, seasonality, trend](https://cloudfabrix.com/blog/aiops/anomaly-detection-time-series-data/) is Ravi Teja Verma's Anomaly Detection and Typical Challenges with Time Series Data, on how to detect anomalies in time-series data with AIOPS. [Rt anomaly detection for time series pinterest](https://medium.com/pinterest-engineering/building-a-real-time-anomaly-detection-system-for-time-series-at-pinterest-a833e6856ddd) using stl decomposition is Pinterest Engineering's post by Kevin Chen and Brian Overstreet. A further STL post, AD, used to be linked here and is kept at the end.

Sliding windows are the other building block. [Solving sliding window problems](https://medium.com/outco/how-to-solve-sliding-window-problems-28d67601a66) is Sergey Piterman's How to Solve Sliding Window Problems, the interview-style problem family that is a subset of dynamic programming. [Rolling window regression](https://medium.com/making-sense-of-data/time-series-next-value-prediction-using-regression-over-a-rolling-window-228f0acae363) applies the window to next-value prediction.

ARIMA residuals are a common detector too. Forecasting using Arima 1 [2](http://alkaline-ml.com/pmdarima/) is pmdarima: ARIMA estimators for Python. Auto arima 1 [2](https://stackoverflow.com/questions/22770352/auto-arima-equivalent-for-python) is the Stack Overflow question auto.arima() equivalent for python, asked because statsmodels had no function for tuning the order (p,d,q) the way R's forecast::auto.arima() does. [3](https://www.analyticsvidhya.com/blog/2018/08/auto-arima-time-series-modeling-python-r/) is Aishwarya Singh's Build High Performance Time Series Models using Auto ARIMA in Python and R, a basic introduction to various time series forecasting methods and techniques.

For statistical tests, [Twitters ESD test](https://medium.com/@elisha_12808/time-series-anomaly-detection-with-twitters-esd-test-50cce409ced1) for outliers, using z-score and t test. Another esd test inside here, kept at the end of the page. Seasonal methods need enough history, so see [Minimal sample size for seasonal forecasting](https://robjhyndman.com/papers/shortseasonal.pdf).

In production, [Golden signals](https://www.usenix.org/conference/srecon19asia/presentation/chen-yu) is Yu Chen's Anomaly Detection on Golden Signals at USENIX, and the talk is on [youtube](https://www.youtube.com/watch?v=3T9ZzQQiPSo) as SREcon19 Asia/Pacific - Anomaly Detection on Golden Signals. Beyond single series, [Graph-based Anomaly Detection and Description: A Survey](https://arxiv.org/pdf/1404.4679.pdf) covers graphs. Finally, Time2vec [paper](https://arxiv.org/pdf/1907.05321.pdf), Time2Vec: Learning a Vector Representation of Time, is Time2vec for deep learning layer.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

\- Clustering time series, subsequences with a rolling window, the pitfall.. This address no longer opens: <https://towardsdatascience.com/dont-make-this-mistake-when-clustering-time-series-data-d9403f39bbb2> - TS anomaly algos (stl, trees, arima). This address no longer opens: <https://blog.statsbot.co/time-series-anomaly-detection-algorithms-1cef5519aef2> - 3. This address no longer opens: <https://towardsdatascience.com/detecting-the-fault-line-using-k-mean-clustering-and-ransac-9a74cb61bb96> - 4. This address no longer opens: <http://www.cs.tau.ac.il/\\~turkel/imagepapers/RANSAC4Dummies.pdf> - here. This address no longer opens: <https://towardsdatascience.com/anomaly-detection-def662294a4e> - Time Series Hierarchical Clustering using Dynamic Time Warping in Python. This address no longer opens: <https://towardsdatascience.com/time-series-hierarchical-clustering-using-dynamic-time-warping-in-python-c8c9edf2fda5> - notebook. This address no longer opens: <https://github.com/avchauzov/\\_articles/blob/master/1.1.trajectoriesClustering.ipynb> - AD. This address no longer opens: <https://medium.com/wwblog/anomaly-detection-using-stl-76099c9fd5a7>


# Fourier Transform

A signal recorded over time often hides its structure in frequency, so before modeling it you need a way to see which waves it is made of. The page starts with the Fourier transform, which decomposes a signal into sine and cosine waves, and then moves to wavelets, which keep track of when those frequencies happen.

The same notes are in [Electronic Network Frequency Analysis](/ai-product/electronic-network-frequency-analysis) and [Feature Engineering](/predictive-ml/audio-feature-engineering).

## Fourier Transform

The Fourier transform is a technique to decompose signals into sine and cosine waves. The applied entry point is Khairul Omar's [Deconstructing time series using FT](https://medium.com/@khairulomar/deconstructing-time-series-using-fourier-transform-e52dd535a44e), which starts from time series as data captured at equally spaced periods, the kind of data that weather measurements, stock markets, mobile transmission, and now Internet of Things devices generate continuously.

The math behind it is a Medium series, mostly on the math, in five parts. Part [1](https://medium.com/sho-jp/fourier-transform-101-part-1-b69ea3cb4837) opens the series. Part [2](https://medium.com/sho-jp/fourier-transform-101-part-2-complex-fourier-series-934a885b3921) recaps the real Fourier series from Part 1, where a periodic signal f(t) is approximated by P(t), and moves to the complex Fourier series. Part [3](https://medium.com/sho-jp/fourier-transform-101-part-3-fourier-transform-6def0bd2ca9b) relates the complex Fourier series to the continuous Fourier transform before stepping into what that transform means. Part [4](https://medium.com/sho-jp/fourier-transform-101-part-4-discrete-fourier-transform-8fc3fbb763f3) is the discrete Fourier transform, and part [5](https://medium.com/sho-jp/fourier-transform-101-part-5-fast-fourier-transform-fft-38c22e05ead3) is the fast Fourier transform (FFT).

## Wavelets

A Fourier view says which frequencies are present but not when, and wavelets are the next step for data science on signals. Two Medium pieces used to open this reading, one on what a wavelet is and how we use it for DS and one on multiple time series classification using continuous wavelet transformation and scalograms; both addresses no longer open and are kept at the end of the page.

The explanation that remains is Shaw Talebi's [Medium on the wavelet transform](https://medium.com/data-science/the-wavelet-transform-e9cfa85d7b34), The Wavelet Transform. The same Medium on the wavelet transform article also sits at its original address, <https://towardsdatascience.com/the-wavelet-transform-e9cfa85d7b34>.

To run wavelets in code, [pywavelets](https://pywavelets.readthedocs.io/en/latest/) — PyWavelets — is open source wavelet transform software for [Python](http://python.org/), the official home of the Python programming language. It combines a simple high level interface with low level C and Cython performance.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Medium on What is a wavelet and how do we use it for DS. This address no longer opens: <https://towardsdatascience.com/what-is-wavelet-and-how-we-use-it-for-data-science-d19427699cef>
* Medium multiple time series classification using continuous wavelet transformation and scalograms. This address no longer opens: <https://towardsdatascience.com/multiple-time-series-classification-by-using-continuous-wavelet-transformation-d29df97c0442>


# Digital Signal Processing (DSP)

Once a signal has been decomposed into frequencies, the practical question is which helpers actually filter it, find its peaks, and detect beats. This page is a toolbox shelf: SciPy for time-domain signal processing and peak finding first, then beat detection with Librosa and a real-time detector, then two modeling notes on activity recognition and self-attention for sound.

The same notes are in [Electronic Network Frequency Analysis](/ai-product/electronic-network-frequency-analysis) and [Feature Engineering](/predictive-ml/audio-feature-engineering).

The base layer is SciPy. [Scipy signal processing](https://docs.scipy.org/doc/scipy/reference/signal.html) is the signal processing module (scipy.signal) in the SciPy v1.18.0 manual, and [Script find peaks](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.find_peaks.html) is its find\_peaks reference, the function to reach for when the question is where a signal spikes.

Peaks in audio lead directly to beat detection. [Real time bpm beat det](https://github.com/shunfu/python-beat-detector) is a Python beat detector that does real-time detection of beats for audio, calculates BPM, and flashes an LED strip in time with music. Librosa also covers time-domain processing and beat detection (and temp), but its original docs addresses no longer open and are kept at the end of the page.

The same time-series thinking carries over to sensors. [Mastery on Human activity recognition, smartphones](https://machinelearningmastery.com/cnn-models-for-human-activity-recognition-time-series-classification/) is MachineLearningMastery.com on 1D convolutional neural network models for human activity recognition from smartphone data.

(Out of place) — Christopher Dossman's [using self-attention for sound signal processing](https://medium.com/ai%C2%B3-theory-practice-business/toward-interpretable-music-tagging-with-self-attention-67a8136048d0) is Toward Interpretable Music Tagging with Self-Attention. Out place on a DSP shelf, it is a model rather than a signal helper.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Librosa. This address no longer opens: <https://librosa.org/doc/latest/core.html#time-domain-processing>
* Beat detection (and temp). This address no longer opens: <https://librosa.org/doc/latest/beat.html#beat>


# Audio Basics

Before engineering audio features or picking an algorithm, you need to know what sound is once it is digitized and why spectrograms sit at the center of audio deep learning. The page follows Ketan Doshi's series in order: sound and spectrograms, Mel spectrograms, data preparation, sound classification, speech recognition, and finally beam search.

Series by Ketan Doshi:

Part 1, [State-of-the-Art Techniques](https://towardsdatascience.medium.com/audio-deep-learning-made-simple-part-1-state-of-the-art-techniques-da1d3dff2504), is the grounding: what is sound and how it is digitized, what problems is audio deep learning solving in our daily lives, and what are spectrograms and why they are all-important. Once the spectrogram is the input, part 2, [Why Mel Spectrograms perform better](https://towardsdatascience.medium.com/audio-deep-learning-made-simple-part-2-why-mel-spectrograms-perform-better-aad889a93505), moves to processing audio data in Python: what are Mel Spectrograms and how to generate them.

The same notes are in [Feature Engineering](/predictive-ml/audio-feature-engineering).

With Mel spectrograms in hand, part 3, [Data Preparation and Augmentation](https://towardsdatascience.medium.com/audio-deep-learning-made-simple-part-3-data-preparation-and-augmentation-24c6e1f6b52), shows how to enhance spectrograms features for optimal performance by hyper-parameter tuning and data augmentation. Part 4, [Sound Classification](https://towardsdatascience.medium.com/audio-deep-learning-made-simple-sound-classification-step-by-step-cebc936bbe5), puts that data to work in an end-to-end example and architecture to classify ordinary sounds, a foundational application for a range of scenarios. Part 5, [Automatic Speech Recognition](https://towardsdatascience.medium.com/audio-deep-learning-made-simple-automatic-speech-recognition-asr-how-it-works-716cfce4c706), turns from sounds to words: voice assistants such as Google Home, Amazon Echo, Siri, and Cortana start from a clip of spoken audio and extract the spoken words as text, and the article walks the Speech-to-Text algorithm and architecture, using CTC Loss and Decoding for aligning sequences.

The same notes are in [Speech](/generative-ai/speech).

Decoding is where the series ends. Part 6, [Beam Search](https://towardsdatascience.medium.com/foundations-of-nlp-explained-visually-beam-search-how-it-works-1586b9849a24), is the algorithm commonly used by Speech-to-Text and NLP applications to enhance predictions.

The same notes are in [Decoding Algorithms For NLP](/natural-language-processing-nlp/decoding-algorithms-for-nlp).


# Audio Terminology

The algorithm pages name their tasks as if the reader already knows them, so the terms come first. The page defines audio source separation, then sound event detection, then query-based separation, which narrows separation down to one sound you ask for.

The same notes are in [Audio Source Separation](/predictive-ml/audio-algorithms#audio-source-separation) and [Sound Event Detection](/predictive-ml/audio-algorithms#sound-event-detection).

Audio source separation is the task of pulling specific sound sources out of an audio mixture. Source separation techniques in audio processing can be classified into various methods, such as blind source separation, supervised source separation, and semi-supervised source separation, and these techniques use algorithms and signal processing methods to extract specific sound sources from an audio mixture. Note that usually these methods are for multi-channel audio; if we have a single-channel audio recording, this is more challenging.

Separation asks what the sources are; sound event detection asks when they happen. Sound Event Detection (SED) is the task of recognizing the sound events and their respective temporal start and end time in a recording. Sound events in real life do not always occur in isolation, but tend to considerably overlap with each other, and recognizing such overlapping sound events is referred to as polyphonic SED.

Query-based separation combines the two ideas by targeting one source. Query-based separation in audio could involve using specific sounds or queries to identify and extract particular sounds from an audio recording. For example, in a crowded audio environment, you might use query-based separation to extract the voice of a specific speaker from a mixture of voices. By providing a query related to the speaker's voice characteristics or specific phrases they are saying, algorithms can be designed to identify and separate that particular speaker's voice from the overall audio recording.


# Audio Feature Engineering

A detection or separation model only sees what the features give it, so audio work starts by turning waves into numbers. The page moves from the Fourier transform to a first read on speech feature extraction, then a Learning from Audio series, then spectrograms, and ends on Mel-frequency cepstral coefficients (MFCC).

The same notes are in [Basics](/predictive-ml/audio-basics), [Digital Signal Processing (DSP)](/predictive-ml/digital-signal-processing-dsp), [Fourier Transform](/predictive-ml/fourier-transform), and [SIGNAL PROCESSING NN (FFT, WAVELETS, SHAPELETS)](/deep-learning/other-architectures#signal-processing-nn-fft-wavelets-shapelets).

Every audio feature below starts from the Fourier transform. DataMListic's Fourier Transform Formula Explained on [YouTube](https://www.youtube.com/watch?v=7Tk6BAJ3mm8) is the short video on the formula itself.

With the transform in place, [Read this first](https://jonathan-hui.medium.com/speech-recognition-feature-extraction-mfcc-plp-5455f5a69dd9), by Jonathan Hui — the speech recognition feature extraction pipeline end to end: A/D, boost, Mel-frequency cepstral coefficients (MFCC), Perceptual Linear Prediction (PLP), Hanning vs Hamming, window, discrete Fourier transform (DFT), IDFT.

The Learning from Audio series by mlearnere then builds the same features one step at a time. It starts with [Learning from Audio: Wave Forms](https://towardsdatascience.medium.com/learning-from-audio-wave-forms-46fc6f87e016), then stays in the time domain with [Learning from Audio: Time Domain Features](https://towardsdatascience.medium.com/learning-from-audio-time-domain-features-4543f3bda34c), which argues that while deep learning often works in the frequency domain, features extracted from the amplitude of each sample still describe the wave form's properties. It crosses into frequency with [Learning from Audio: Fourier Transformation](https://towardsdatascience.medium.com/learning-from-audio-fourier-transformations-f000124675ee), reaches [Learning from Audio: The Mel Scale, Mel Spectrograms, and Mel Frequency Cepstral Coefficients](https://towardsdatascience.medium.com/learning-from-audio-the-mel-scale-mel-spectrograms-and-mel-frequency-cepstral-coefficients-f5752b6324a8), and ends with [Learning from Audio: Pitch and Chromagrams](https://towardsdatascience.medium.com/learning-from-audio-pitch-and-chromagrams-5158028a505), which goes beyond frequency over time once the spectrogram and MFCCs are understood.

The spectrogram is the feature most audio models actually consume, so it gets its own reading. [Understanding the Mel Spectrogram](https://medium.com/analytics-vidhya/understanding-the-mel-spectrogram-fca2afa2ce53) is the first explanation. [2](https://towardsdatascience.medium.com/getting-to-know-the-mel-spectrogram-31bca3e2d9d0) is Getting to Know the Mel Spectrogram: read this short post if you want to be like Neo and know all about the Mel Spectrogram, or maybe not all, but at least a little. [3](https://medium.com/hacking-media/beginner-guide-to-visualizing-audio-as-a-spectogram-in-python-65dca2ab1e61) is Thomas Christiansen's beginner's guide to visualizing audio as a spectrogram in Python, treating audio as something to see rather than only hear. [4](https://importchris.medium.com/how-to-create-understand-mel-spectrograms-ff7634991056) is Christopher Lewis on how to create and understand Mel-spectrograms, where the horizontal axis is time, the vertical axis is frequency, and color intensity is the amplitude of a frequency at a point in time. Frames of a spectrogram also change over time, and DataMListic's Spectral Features - Deltas and Delta-Deltas Explained, [delta deltas](https://www.youtube.com/watch?v=zxEnuPolylY), covers those first and second differences.

The last step compresses the Mel spectrogram into Mel-frequency cepstral coefficients (MFCC). DataMListic's Mel Frequency Cepstral Coefficients (MFCC) Explained on [YouTube](https://www.youtube.com/watch?v=SJo7vPgRlBQ) is the video for it.


# Audio Algorithms

Once the task names and the features are clear, the remaining question is which model or package actually does the job. The page goes from sound event detection to query-based and open-domain source separation, then blind source separation, then embeddings and other pretrained tools, and ends with toolkits and write-ups for classification, speech recognition, and annotation.

## Sound Event Detection

Detection comes first because it is the most direct use of a pretrained audio model: say which sound events occur and when.

The same notes are in [Terminology](/predictive-ml/audio-terminology).

[YamNet](https://github.com/tensorflow/models/tree/master/research/audioset/yamnet) sits in the TensorFlow models repository of models and examples built with TensorFlow, and it is the pretrained model the rest of this section builds on. The robertanto repository on [GitHub](https://github.com/robertanto/Real-Time-Sound-Event-Detection) is the Python implementation of a sound event detection system working in real time, built on YamNet, and real-time sound event detection is exactly what it is for. Its [Event types labels list](https://github.com/robertanto/Real-Time-Sound-Event-Detection/blob/main/keras_yamnet/yamnet_class_map.csv) is the YamNet class map used by that real time sound event detection code; the relevant labels are 420 430.

## Query-based separation

Detection says a sound is there; query-based separation pulls that one sound out of the mix when you give an example of it.

[Zero Shot Audio Source Separation](https://github.com/RetroCirce/Zero_Shot_Audio_Source_Separation) is the official code repo for the AAAI 2022 work, the [paper](https://arxiv.org/abs/2112.07891) is "Zero-shot Audio Source Separation through Query-based Learning from Weakly-labeled Data" on arXiv, and the [interface](https://replicate.com/retrocirce/zero_shot_audio_source_separation) on Replicate runs zero shot sound separation by arbitrary query samples — a three-component pipeline that lets you train an audio source separator to separate any source from the track. All you need is a mixture audio to separate, and a given source sample as a query. Then the model will separate your specified source from the track.

## Audio Source Separation

Querying by a sample is one way to pick a source; the models here separate open-domain sources, including by natural language, or learn separation directly on the waveform.

The same notes are in [Terminology](/predictive-ml/audio-terminology).

[Audio Sep](https://github.com/Audio-AGI/AudioSep) — AudioSep is a foundation model for open-domain sound separation with natural language queries. AudioSep demonstrates strong separation performance and impressive zero-shot generalization ability on numerous tasks such as audio event separation, musical instrument separation, and speech enhancement.

The waveform route is Wave-U-Net, and it comes in several implementations. The [Original](https://github.com/f90/Wave-U-Net) is f90/Wave-U-Net, the implementation of the Wave-U-Net for audio source separation. The [Pytorch](https://github.com/f90/Wave-U-Net-Pytorch) version is the improved Wave-U-Net implemented in Pytorch. [TF2 / Keras](https://github.com/satvik-venkatesh/Wave-U-net-TF2) is satvik-venkatesh/Wave-U-net-TF2, which implements the Wave-U-net architecture in TensorFlow 2. [For speech enhancements](https://github.com/craigmacartney/Wave-U-Net-For-Speech-Enhancement) is improved speech enhancement with the Wave-U-Net, a deep convolutional neural network architecture for audio source separation, implemented for the task of speech enhancement in the time-domain.

## Blind Source Separation

The models above learn what sources sound like from labeled data; blind source separation works without that, from priors or from the spatial structure of multi-channel recordings.

[Deep Audio Prior](https://github.com/adobe/Deep-Audio-Prior) — Our deep audio prior can enable several audio applications: blind sound source separation, interactive mask-based editing, audio textual synthesis, and audio watermarker removal.

BSS ([EM source separation](https://github.com/fgnt/pb_bss)) — This repository covers EM algorithms to separate speech sources in multi-channel recordings. In particular, the repository contains methods to integrate Deep Clustering (a neural network-based source separation algorithm) with a probabilistic spatial mixture model as proposed in the Interspeech paper "Tight integration of spatial and spectral features for BSS with Deep Clustering embeddings" presented at Interspeech 2017 in Stockholm.

## Image embeddings and others

Separation is not the only downstream task; many audio pipelines need a pretrained representation or a single-purpose detector instead.

The same notes are in [Embedding](/deep-learning/representations).

[Openl3](https://github.com/marl/openl3) — OpenL3: Open-source deep audio and image embeddings. For pitch, [Pitch estimation](https://github.com/marl/crepe) is marl/crepe, CREPE: A Convolutional REpresentation for Pitch Estimation, a pre-trained model (ICASSP 2018).

[Speaker recognition](https://github.com/Anwarvic/Speaker-Recognition) — Speaker recognition is the identification of a person given an audio file. It is used to answer the question "Who is speaking?" Speaker verification (also called speaker authentication) is similar to speaker recognition, but instead of returning the speaker who is speaking, it returns whether the speaker (who is claiming to be a certain one) is truthful or not. Speaker Verification is considered to be a little easier than speaker recognition.

Before any of these runs, a pipeline often needs to know whether anyone is speaking at all. [Voice activity detector](https://github.com/snakers4/silero-vad) is snakers4/silero-vad, Silero VAD, a pre-trained enterprise-grade Voice Activity Detector. The list of pretrained audio models these tools were drawn from is taken from [here](https://www.mathworks.com/help/audio/referencelist.html?type=function\&category=pretrained-models\&s_tid=CRUX_topnav), the MathWorks pretrained-model list.

## Other Tools

Beyond single models, the remaining shelf is full toolkits and write-ups for separation, classification, speech recognition, and annotation.

The same notes are in [Deep Neural Audio](/deep-learning/deep-neural-audio) and [Speech](/generative-ai/speech).

[KALDI](https://kaldi-asr.org/models.html) speech recognition toolkit with many SOTA models is the toolkit to start from.

Separation also has an end-to-end story in the Audio AI series by Alejandro Koretzky. [Isolating instruments from stereo music using Convolutional Neural Networks](https://towardsdatascience.medium.com/audio-ai-isolating-vocals-from-stereo-music-using-convolutional-neural-networks-210532383785) is the first article, on isolating vocals from stereo music: a medium quality mp3 of We Can Work it Out by The Beatles goes in, and the isolated vocals come out of the model. [part 2](https://towardsdatascience.medium.com/audio-ai-isolating-instruments-from-stereo-music-using-convolutional-neural-networks-584ababf69de) is the follow-up on isolating instruments, which recommends reading the vocal isolation article first.

For classification, Mike Smales' Udacity capstone on classifying urban sounds walks the full path: [Sound classification using CNN, loading and normalizing sounds using librosa, converting to a 2d spectrogram image, using CNN on top.](https://medium.com/@mikesmales/sound-classification-using-deep-learning-8bc2aa1990b7)

For speech, [Speech recognition with DL](https://medium.com/@ageitgey/machine-learning-is-fun-part-6-how-to-do-speech-recognition-with-deep-learning-28293c162f7a) — how to convert sounds to vectors, feeding into an RNN. (Great) [Jonathan Hui on speech recognition](https://medium.com/@jonathan_hui/speech-recognition-series-71fd6784551a) is a Great series that covers the basics, like phonetics, and the machine learning models used in speech recognition, and then applies deep learning to it.

Speech models also need labeled conversations. [Gecko](https://medium.com/gong-tech-blog/introducing-gecko-an-open-source-solution-for-effective-annotation-of-conversations-2ecec0909941) — ([github.com/gong-io/gecko](https://github.com/gong-io/gecko)) [YouTube](https://www.youtube.com/watch?v=CBYA0YC1NBI), is an open-source tool for the annotation of the linguistic content of conversations, introduced by Golan Levy on the Gong tech blog, where automatic speech recognition is at the core of the product. It can be used for segmentation, diarization, and transcription. With Gecko, you can create and perfect audio-based datasets, compare the results of multiple models simultaneously, and highlight differences between transcriptions.


# Graph Theory

When data is a set of relationships rather than a table, the questions become how nodes connect, which nodes matter, and which groups form. The page walks the graph topics first, from general libraries through connectivity, community detection, and node2vec embeddings to how communities are evaluated, then the Neo4j centrality, community, and experimental algorithms, and ends with graph and GNN courses and the Python tools to run it all.

The same notes are in [Block Modeling - Distance Matrices](/predictive-ml/clustering-algorithms#block-modeling---distance-matrices), [Deep walk](/deep-learning/graph-neural-nets#deep-walk), [Fraud Detection](/ai-product/fraud-detection), [GRAPH NEURAL NETWORKS (GNN)](/deep-learning/graph-neural-nets#graph-neural-networks-gnn), [Root Cause Effects (RCE/RCA)](/decision-intelligence/root-cause-effects-rce-rca), and [Social Network Analysis](/predictive-ml/social-network-analysis).

## Graph Topics

The general-purpose starting point is a single library that covers many unsupervised graph methods at once. [General purpose and community detection GIT](https://github.com/benedekrozemberczki/karateclub) is Karate Club, an API oriented open-source Python framework for unsupervised learning on graphs (CIKM 2020).

Connectivity is the first structural question: where does the graph come apart? Min-cut: [1](https://github.com/gsw73/min-cut/blob/master/karger_min_cut.py) implements the Karger min cut algorithm on a graph read from a text file. [2](https://github.com/ChuntaoLu/Algorithms-Design-and-Analysis/blob/master/week3%20Karger%20min%20cut/min_cut.py) is the Karger min cut from a repository of algorithms learned in Stanford's Algorithms: Design and Analysis. Min cut [3](https://github.com/WithaK16/kargerMinCut/blob/master/kargerMinCut.py) is another Karger min cut algorithm implementation in Python.

Cutting a graph leads directly to finding its communities. [Louvain community](https://github.com/taynaud/python-louvain/) is taynaud/python-louvain, Louvain community detection. Girvan–Newman [gist](https://gist.github.com/chelsea1992/6c725a24d358763097bebe8223c2014a) is community detection with the Girvan-Newman algorithm, which several repositories implement as divisive hierarchical clustering: [this worked](https://github.com/ZwEin27/Community-Detection) is ZwEin27/Community-Detection, and [this is potentially good too](https://github.com/riteshkasat/Community-Detection-Algorithm) is riteshkasat/Community-Detection-Algorithm. The remaining implementations are [another](https://github.com/ServiceCutter/girvan-newman), the Girvan-Newman clustering used by the Service Cutter; [another](https://github.com/ZwEin27/Community-Detection), the same ZwEin27 repository again; and [another](https://github.com/kjahan/community), kjahan's Python implementation of the Girvan-Newman algorithm.

Communities describe groups; embeddings give each node a vector a model can use. [Node2vec](https://github.com/eliorc/Medium/blob/master/Nod2Vec-FIFA17-Example.ipynb) is the FIFA17 example notebook in eliorc's repository of code related to his Medium blog posts. The [paper](https://arxiv.org/pdf/1607.00653.pdf) is node2vec: Scalable Feature Learning for Networks. [medium1](https://towardsdatascience.medium.com/think-your-data-different-ddc435f70850) is Think your Data Different, on applying deep learning to graph datasets in social networks, recommender systems, and biology, where data is inherently structured as a graph. [medium 2](https://towardsdatascience.medium.com/node2vec-embeddings-for-graph-data-32a866340fef) is node2vec: Embeddings for Graph Data, which argues that quality embeddings are the opposite of "garbage in, garbage out". The implementation of the node2vec algorithm — tutorial — is the [code](https://github.com/eliorc/node2vec), and the [git code](https://github.com/eliorc/Medium/blob/master/Nod2Vec-FIFA17-Example.ipynb) is the same FIFA17 notebook from the Medium posts. The [original py2 code](https://github.com/aditya-grover/node2vec) is aditya-grover/node2vec, and [taboola code for their medium paper](https://github.com/taboola/node2vec-example/blob/master/node2vec.ipynb) is the taboola/node2vec-example notebook.

The same notes are in [Node2vec](/deep-learning/graph-neural-nets#node2vec).

Once communities are found, they have to be judged. [Evaluation metrics for community detection](https://stackoverflow.com/questions/28952104/evaluation-metrics-for-community-detection-algorithms) is the Stack Overflow question on evaluation metrics for community detection algorithms. [Review for community detection algorithms](https://arxiv.org/pdf/0906.0612.pdf) is Community detection in graphs, and the same review's abstract page is the [paper](https://arxiv.org/abs/0906.0612): community structure, or clustering, is the organization of vertices in clusters with many edges joining vertices of the same cluster and comparatively few edges joining vertices of different clusters. [Term: community structure](https://en.wikipedia.org/wiki/Community_structure#Algorithms_for_finding_communities) is the Wikipedia entry on community structure and its algorithms for finding communities, and \[Term: modularity of networks]\(<https://en.wikipedia.org/wiki/Modularity\\_(networks)> is the Wikipedia entry on modularity (networks).

Two papers are still on the reading pile. [Unread paper](http://science.sciencemag.org/content/328/5980/876) is the first. [Unread comparison of community detection algos](https://arxiv.org/abs/1406.2205) compares community detection algorithms for multiplex graphs, where a multiplex is a set of graphs on the same vertex set, i.e. a generalized graph for multiple relationships with parallel edges between vertices.

Community detection can also be treated as clustering a matrix. [Clustering adjacency matrices](https://stats.stackexchange.com/questions/125295/the-best-way-for-clustering-an-adjacency-matrix) is Fahd's Cross Validated question on the best way for clustering an adjacency matrix, built from partial correlations of neural time series. [Spectral-clustering](https://calculatedcontent.com/2012/10/09/spectral-clustering/) is Spectral Clustering: A quick overview, from calculated | content. [Finding natural groups in undirected graphs](https://stats.stackexchange.com/questions/142297/finding-natural-groups-clusters-in-an-undirected-graph-over-several-undirect) is the question on finding natural groups or clusters in one undirected graph or over several graphs that may share nodes. For more, [Awesome community detection on github](https://github.com/benedekrozemberczki/awesome-community-detection) is a curated list of community detection research papers with implementations.

The ready-made versions of these algorithms are in the Neo4j Graph Data Science library, and [Various algorithms](https://neo4j.com/docs/graph-algorithms/current/algorithms/closeness-centrality/) is its manual; the subsections below follow its chapters.

### Centrality algorithms

Centrality answers which nodes matter most, and [5. Centrality algorithms](https://neo4j.com/docs/graph-algorithms/current/algorithms/centrality/) is the Neo4j chapter with explanations and examples for each centrality algorithm in the library:

* [5.1. The PageRank algorithm](https://neo4j.com/docs/graph-algorithms/current/algorithms/page-rank/)
* [5.2. The Betweenness Centrality algorithm](https://neo4j.com/docs/graph-algorithms/current/algorithms/betweenness-centrality/)
* [5.3. The Closeness Centrality algorithm](https://neo4j.com/docs/graph-algorithms/current/algorithms/closeness-centrality/)
* [5.4. The Degree Centrality algorithm](https://neo4j.com/docs/graph-algorithms/current/algorithms/degree-centrality/)

### Community detection algorithms

After the important nodes come the groups, and [6. Community detection algorithms](https://neo4j.com/docs/graph-algorithms/current/algorithms/community/) is the chapter with explanations and examples for each community detection algorithm in the library, including Louvain from the topics above:

* [6.1. The Louvain algorithm](https://neo4j.com/docs/graph-algorithms/current/algorithms/louvain/)
* [6.2. The Label Propagation algorithm](https://neo4j.com/docs/graph-algorithms/current/algorithms/label-propagation/)
* [6.3. The Connected Components algorithm](https://neo4j.com/docs/graph-algorithms/current/algorithms/connected-components/)

### Experimental algorithms

Beyond the stable chapters, [7. Experimental algorithms](https://neo4j.com/docs/graph-algorithms/current/experimental-algorithms/) extends the same manual to procedures, path finding, similarity, and link prediction:

* [7.1. Procedures](https://neo4j.com/docs/graph-algorithms/current/experimental-procedures/)
* [7.2. Centrality algorithms](https://neo4j.com/docs/graph-algorithms/current/experimental-algorithms/centrality/)
* [7.3. Community detection algorithms](https://neo4j.com/docs/graph-algorithms/current/experimental-algorithms/community/)
* [7.4. Path finding algorithms](https://neo4j.com/docs/graph-algorithms/current/experimental-algorithms/pathfinding/)
* [7.5. Similarity algorithms](https://neo4j.com/docs/graph-algorithms/current/experimental-algorithms/similarity/)
* [7.6. Link Prediction algorithms](https://neo4j.com/docs/graph-algorithms/current/experimental-algorithms/linkprediction/)
* [7.7. Preprocessing functions and](https://neo4j.com/docs/graph-algorithms/current/experimental-algorithms/preprocessing/)

## Graph/GNN courses

The algorithms above are classical; the courses carry the same graphs into machine learning and graph neural networks.

The same notes are in [GNN courses](/deep-learning/graph-neural-nets#gnn-courses).

[Machine learning with graphs by Stanford](http://web.stanford.edu/class/cs224w/) is CS224W. [Graph deep learning course](https://geometricdeeplearning.com/lectures/) is the geometric deep learning course on grids, groups, graphs, geodesics, and gauges. Its keynote version is Michael Bronstein's ICLR 2021 talk "Geometric Deep Learning: The Erlangen Programme of ML" — graphs, sets, groups, GNNs — on [YouTube](https://www.youtube.com/watch?v=w6Pw4MOzMuo).

## Graph Tools

To run any of this in Python, two libraries do the graph handling. [Graph-tool](https://graph-tool.skewed.de/) is an efficient Python module for manipulation and statistical analysis of graphs. [NetworkX](https://github.com/networkx/networkx) is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.


# Social Network Analysis

Graph theory gives the machinery; social network analysis asks what that machinery says about people who are connected to each other. The page starts with definitions and the people who shaped the field, then the core measures, then how traits spread through networks and why that is hard to prove, and ends with the tools and an applied walkthrough.

The same notes are in [Graph Theory](/predictive-ml/graph-theory).

The definition comes first: the [Wiki](https://en.wikipedia.org/wiki/Social_network) entry is the Wikipedia article on the social network. The algorithmic view of the same object is the [Paper: algorithmic approach to social networks](http://web.archive.org/web/20240712194953/http://www.cs.carleton.edu/faculty/dlibenno/papers/thesis/thesis.pdf), kept here through its archived copy. [Steve Borgatti](https://sites.google.com/site/steveborgatti/home) is the home page of one of the people behind social network analysis methods.

From definitions to measures, [Intro to SNA](http://www.orgnet.com/sna.html) is Orgnet's Social Network Analysis: An Introduction, from a network analysis consulting, training, and software firm. It walks centrality, betweenness centrality, network centralization, network reach, network integration, boundary spanners, and peripheral players. Whether more connections can make up for weaker ones is the question in [Social Network Analysis: Can Quantity Compensate for Quality?](https://33bits.wordpress.com/2009/02/15/social-network-analysis-can-quantity-substitute-for-quality/) on 33 Bits of Entropy.

Measures describe a network; the harder question is why connected people end up alike. [Nicholas Christakis](http://web.archive.org/web/20110603105212/http://www.wjh.harvard.edu/soc/faculty/christakis/) of Harvard and [James Fowler](http://web.archive.org/web/20150810035527/http://jhfowler.ucsd.edu/) of UC San Diego have produced a series of ground-breaking papers analyzing the spread of various traits in social networks: [obesity](http://content.nejm.org/cgi/content/full/357/4/370), [smoking](http://content.nejm.org/cgi/content/full/358/21/2249), [happiness](http://www.bmj.com/cgi/content/full/337/dec04_2/a2338), and most recently, in collaboration with John Cacioppo, [loneliness](http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1319108). The Christakis-Fowler collaboration has now become [well-known](http://web.archive.org/web/20170808104848/http://jhfowler.ucsd.edu/science_friendship_as_a_health_factor.pdf), but from a technical perspective, what was special about their work?

It turns out that they found a way to distinguish between the three reasons why people who are related in a social network are similar to each other. Homophily is the tendency of people to seek others who are alike. For example, most of us restrict our dates to smokers or non-smokers, mirroring our own behavior. Confounding is the phenomenon of related individuals developing a trait because of a (shared) environmental circumstance. For example, people living right next to a McDonald’s might all gradually become obese. Induction is the process of one individual passing a trait or behavior on to their friends, whether by active encouragement or by setting an example.

To compute any of these measures on real data, [Networkx](https://networkx.org/documentation/networkx-1.10/reference/algorithms.html) is the algorithms reference — Centrality is just a fraction of the algorithms contained in networkx, as the figure of its algorithm list shows.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-f63364ed55fc1ffe7bdec570fcd95cdbb7b77658%2Fgimg-b4faaaed3b79.png?alt=media" alt=""><figcaption><p>Networkx algorithms.</p></figcaption></figure>

The applied walkthrough that ties theory to that code is [Social Network analysis from theory to applications](https://towardsdatascience.medium.com/social-network-analysis-from-theory-to-applications-with-python-d12e9a34c2c7), which introduces the theory of social networks with a short introduction to graph theory and information spread, then dives into Python code with NetworkX that constructs social networks from real datasets. It is by [Dima Goldenberg](https://www.linkedin.com/in/dimgold/), a machine learning group leader at Booking.com.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* Paper: algorithmic approach to social networks. This address no longer opens: <http://www.cs.carleton.edu/faculty/dlibenno/papers/thesis/thesis.pdf>
* Nicholas Christakis. This address no longer opens: <http://www.wjh.harvard.edu/soc/faculty/christakis/>
* James Fowler. This address no longer opens: <http://jhfowler.ucsd.edu/>
* well-known (science friendship as a health factor PDF). This address no longer opens: <http://jhfowler.ucsd.edu/science\\_friendship\\_as\\_a\\_health\\_factor.pdf>




---

[Next Page](https://www.mlcompendium.com/llms-full.txt/1)