> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/decision-intelligence/a-b-testing.md).
# A/B Testing
A product change only counts if an online controlled experiment shows it moved the metric that matters. The page starts with the book and course that teach the method, then CUPED and the other variance-reduction work that makes tests shorter, then the practical guides, and only after that the tools for running tests.
The same notes are in [Contextual Bandits](/decision-intelligence/contextual-bandits.md), [Drift](/ai-engineering/mlops-monitoring-and-alerts.md#drift), [Hypothesis Testing](/decision-intelligence/hypothesis-testing.md), [Multi Armed Bandits](/decision-intelligence/multi-armed-bandits.md), and [Tutorials](/ai-engineering/full-stack-and-ops-1/tutorials.md).
The reading starts with the one book to own. (highly recommended — buy) [Trustworthy controlled experiments](https://www.amazon.com/Trustworthy-Online-Controlled-Experiments-Practical/dp/1108724264) is the book on running controlled experiments you can trust; highly recommended, buy it, Ron and Diane's book. For a guided path instead of a book, (really good) [A comprehensive A/B testing course](https://www.dynamicyield.com/lesson/introduction-to-ab-testing/) is the course the note credits to dynamic fields, starting from an introduction to A/B testing, and it is really good.
Once the basic test is clear, the next problem is sensitivity: a test that needs fewer users to see the same effect. CUPED — [Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data](https://www.researchgate.net/publication/237838291_Improving_the_Sensitivity_of_Online_Controlled_Experiments_by_Utilizing_Pre-Experiment_Data) is the paper that names the method. CUPED is an acronym for ***C**ontrolled experiments **U**sing **P**re-**E**xperiment **D**ata* \[1]. It is a method that aims to estimate treatment effects in A/B tests more accurately than simple differences in means, where we traditionally use the observed difference between the sample means. The paper is by [Ron Kohavi](https://www.researchgate.net/profile/Ron-Kohavi), [Alex Deng](https://www.researchgate.net/profile/Alex-Deng-2), [Ya Xu](https://www.researchgate.net/scientific-contributions/Ya-Xu-2010104340), [Toby Walker](https://www.researchgate.net/profile/Toby-Walker).
Before going deeper into variance, two pieces widen what an experiment can be. (unread) [A/B testing](https://booking.ai/theres-more-to-experimentation-than-a-b-223fba846876) is a further A/B testing piece, still unread. [Bayesian A/B testing](https://towardsdatascience.medium.com/why-you-should-switch-to-bayesian-a-b-testing-364557e0af1a) is Itamar Faran's Why You Should Switch to Bayesian A/B Testing, written for anyone who swore off statistics, its distributions and critical values, until they had to analyze an A/B test. (YouTube) [Lukas Vermeer: Democratizing Online Controlled Experiments at Booking.com — ELITE CAMP 2018](https://www.youtube.com/watch?v=z81SxsmSfFo) is the CXL talk on opening experimentation to a whole company.
At scale the problems stop being about one test. (Paper) [Top Challenges from the first Practical Online Controlled Experiments Summit](https://www.kdd.org/exploration_files/June_2019_-_2.TopChallengesInPracticalOnlineControlledExperiments_.pdf) is the paper written by the summit's participants from Microsoft, Google, LinkedIn, Airbnb, Facebook, Lyft, Netflix and the other organizations its abstract lists:
> Online controlled experiments (OCEs), also known as A/B tests, have become ubiquitous in evaluating the impact of changes made to software products and services. While the concept of online controlled experiments is simple, there are many practical challenges in running OCEs at scale. To understand the top practical challenges in running OCEs at scale and encourage further academic and industrial exploration, representatives with experience in large-scale experimentation from thirteen different organizations (Airbnb, Amazon, Booking.com, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, Yandex, and Stanford University) were invited to the first Practical Online Controlled Experiments Summit. All thirteen organizations sent representatives. Together these organizations have tested more than one hundred thousand experiment treatments last year. Thirty-four experts from these organizations participated in the summit in Sunnyvale, CA, USA on December 13-14, 2018. While there are papers from individual organizations on some of the challenges and pitfalls in running OCEs at scale, this is the first paper to provide the top challenges faced across the industry for running OCEs at scale and some common solutions
The same causal question is older than software. (book) [Modern epidemiology](https://www.stata.com/bookstore/modern-epidemiology/) is Modern Epidemiology, Fourth Edition, which provides a complete desk reference of the methods of modern epidemiology, with contributions from the 4 authors plus 35 leading experts from various subfields.
Back in software, the hard part is reading the results. [Challenges, Best Practices and Pitfalls in Evaluating Results of Online Controlled Experiments](https://sites.google.com/view/kdd2019-exp-evaluation/) is the tutorial on exactly that. A/B Testing is the gold standard to estimate the causal relationship between a change in a product and its impact on key outcome measures. It is widely used in the industry to test changes ranging from simple copy change or UI change to more complex changes like using machine learning models to personalize user experience. The key aspect of A/B testing is evaluation of experiment results. Designing the right set of metrics — correct outcome measures, data quality indicators, guardrails that prevent harm to business, and a comprehensive set of supporting metrics to understand the “why” behind the key movements is the #1 challenge practitioners face when trying to scale their experimentation program. On the technical side, improving sensitivity of experiment metrics is a hard problem and an active research area, with large practical implications as more and more small and medium size businesses are trying to adopt A/B testing and suffer from insufficient power. In this tutorial we will discuss challenges, best practices, and pitfalls in evaluating experiment results, focusing on both lessons learned and practical guidelines as well as open research questions.
The tutorial's slides are two PDFs:
1. Online Metrics Tutorial - [Part I. Introduction](https://drive.google.com/file/d/13rq3KeqCdsM5EiNcr7jWbphc0wWw12sX/view?usp=sharing)
2. Online Metrics Tutorial - [Part II. Best Practices](https://drive.google.com/file/d/1q3uf0Voe0oi7aPyxpPoqYxL2Ie_Vtqnk/view?usp=sharing)
That tutorial's point about insufficient power is where CUPED returns in practice. [Increasing experimentation accuracy and speed by using control variates](https://codeascraft.com/2021/06/02/increasing-experimentation-accuracy-and-speed-by-using-control-variates/) is Etsy's account. In this article, we share details about our team’s journey to bring the statistical method known as CUPED to Etsy, and how it is now helping other teams make more informed product decisions, as well as shorten the duration of their experiments by up to 20%. We offer some perspectives on what makes such a method possible, what it took us to implement it at scale, and what lessons we have learned along the way.
Randomizing users is not always possible. (Microsoft) [Why Tenant-Randomized A/B Test is Challenging and Tenant-Pairing May Not Work](https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/why-tenant-randomized-a-b-test-is-challenging-and-tenant-pairing-may-not-work/) is Microsoft Research on tenant- or enterprise-randomized tests: why tenant-randomization is challenging, the solutions available, and why tenant-pairing may not work.
The clearest single CUPED walkthrough is (good) [How to Double A/B Testing Speed with CUPED](https://towardsdatascience.medium.com/how-to-double-a-b-testing-speed-with-cuped-f80460825a90) — Microsoft’s variance reduction that’s becoming industry standard. It opens with where the method came from and who adopted it:
> Controlled-experiment Using Pre-Existing Data (CUPED) is a variance reduction technique created by [Microsoft in 2013](https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf). Since then, it has been implemented at [Netflix](https://www.kdd.org/kdd2016/papers/files/adp0945-xieA.pdf), [Booking.com](https://booking.ai/how-booking-com-increases-the-power-of-online-experiments-with-cuped-995d186fff1d), [BBC](https://medium.com/bbc-data-science/increasing-experiment-sensitivity-through-pre-experiment-variance-reduction-166d7d00d8fd), and many others.
In short, CUPED uses pre-experiment data to control for natural variation in an experiment’s north star metric. By removing natural variation, we can run statistical tests that require a smaller sample size. CUPED can be added to virtually any A/B testing framework; it’s computationally efficient and fairly straightforward to code.
The same notes are in [SAMPLE SELECTION](/data/datasets-intro/datasets.md#sample-selection).
The companies named in that quote each wrote up their own version. (Netflix) [Improving the Sensitivity of Online Controlled Experiments: Case Studies at Netflix](https://www.kdd.org/kdd2016/papers/files/adp0945-xieA.pdf) treats sensitivity as a controlled experiment's ability to detect differences in business metrics due to product changes. (Booking) [How Booking.com increases the power of online experiments with CUPED](https://booking.ai/how-booking-com-increases-the-power-of-online-experiments-with-cuped-995d186fff1d) is Booking's version. A [comment](https://medium.com/@drsimonj/yes-cuped-could-be-used-like-regression-methods-to-adjust-for-confounders-71c951234fcf) on it adds the caveat: yes, CUPED could be used like regression methods to adjust for confounders, but random assignment lets you make reasonably confident claims about causality, and observational data cannot give the same confidence. (BBC) [Increasing experimental power with variance reduction at the BBC](https://medium.com/bbc-data-science/increasing-experiment-sensitivity-through-pre-experiment-variance-reduction-166d7d00d8fd) is Frank Hopkins' article; it discusses how the Experimentation team have been accounting for pre-experiment variance in order to increase the statistical power of their experiments. (TripAdvisor) (good) [Reducing A/B test measurement variance by 30%+](https://www.tripadvisor.com/engineering/reducing-a-b-test-measurement-variance-by-30/) is the tripadvisor.com engineering write-up of the same goal.
Variance reduction still ends in a statistical test, so the practical guides come next. (medium) [practitioner guide to](https://vkteam.medium.com/practitioners-guide-to-statistical-tests-ed2d580ef04f) is the Practitioner’s Guide to Statistical Tests. From Hubspot, [How to Do A/B Testing: 15 Steps for the Perfect Split Test](https://blog.hubspot.com/marketing/how-to-do-a-b-testing) is for anyone who wants to discover what marketing really works for their audience, and [Sample size calculation](https://blog.hubspot.com/marketing/email-a-b-test-sample-size-testing-time) shows how to calculate the ideal sample size so results are statistically significant and the time frame is accurate. The same guide, filed as [Statistical tests](https://vkteam.medium.com/practitioners-guide-to-statistical-tests-ed2d580ef04f), is the one to return to when choosing the test. Vidhya on the topic has two pieces: [A/B Testing Measurement Frameworks — Every Data Scientist Should Know](https://www.analyticsvidhya.com/blog/2021/03/a-b-testing-measurement-frameworks%E2%80%8A-%E2%80%8Aevery-data-scientist-should-know/) is Mainak's article on the measurement frameworks behind an A/B test; [A/B Testing for Data Science using Python – A Must-Read Guide for Data Scientists](https://www.analyticsvidhya.com/blog/2020/10/ab-testing-data-science/) is Shipra Saxena on A/B Testing as a way to test your products and how data scientists leverage it. (Yandex) [Online Evaluation for Effective Web Service Development](https://research.yandex.com/tutorials/online-evaluation/kdd-2018) is a tutorial at the conference KDD 2018, for engineers, designers, analysts, and managers who want web service development to be effectively data-driven.
Online evaluation for effective web service development.
\## A/B Testing Tools
Once the method and the CUPED reading are in place, the remaining question is what runs the test. The [R-shiny Tool](https://towardsdatascience.medium.com/cuped-r-shiny-tool-7a4869d77f0a) is Frank Hopkins' CUPED R-Shiny Tool, built for controlling the variance that exists within an experimental metric using a strongly correlated pre-experiment metric (covariate). For a full platform, [Wasabi open source platform](http://web.archive.org/web/20240225062855/https://www.intuit.com/blog/innovative-thinking/meet-wasabi-an-open-source-ab-testing-platform/) is Lucian Lita's Meet Wasabi, an Open Source A/B Testing Platform on the Intuit Blog, a real-time, enterprise-grade A/B Testing platform battle-tested in FinTech products. Its [Git](https://github.com/intuit/wasabi) repo, intuit/wasabi, now says the Wasabi A/B Testing service is an open source project that is no longer under active development or being supported. On the commercial side, [Optimizely](https://www.optimizely.com/) sells everything marketing and digital teams need to create content fast, test and personalize, and prove business impact.
## Deprecated links
{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}
* Wasabi open source platform. This address no longer opens:
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://www.mlcompendium.com/decision-intelligence/a-b-testing.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
%20(1).png?alt=media)