A/B Testing
A product change only counts if an online controlled experiment shows it moved the metric that matters. The page starts with the book and course that teach the method, then CUPED and the other variance-reduction work that makes tests shorter, then the practical guides, and only after that the tools for running tests.
The same notes are in Contextual Bandits, Drift, Hypothesis Testing, Multi Armed Bandits, and Tutorials.
The reading starts with the one book to own. (highly recommended — buy) Trustworthy controlled experiments is the book on running controlled experiments you can trust; highly recommended, buy it, Ron and Diane's book. For a guided path instead of a book, (really good) A comprehensive A/B testing course is the course the note credits to dynamic fields, starting from an introduction to A/B testing, and it is really good.
Once the basic test is clear, the next problem is sensitivity: a test that needs fewer users to see the same effect. CUPED — Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data is the paper that names the method. CUPED is an acronym for Controlled experiments Using Pre-Experiment Data [1]. It is a method that aims to estimate treatment effects in A/B tests more accurately than simple differences in means, where we traditionally use the observed difference between the sample means. The paper is by Ron Kohavi, Alex Deng, Ya Xu, Toby Walker.
Before going deeper into variance, two pieces widen what an experiment can be. (unread) A/B testing is a further A/B testing piece, still unread. Bayesian A/B testing is Itamar Faran's Why You Should Switch to Bayesian A/B Testing, written for anyone who swore off statistics, its distributions and critical values, until they had to analyze an A/B test. (YouTube) Lukas Vermeer: Democratizing Online Controlled Experiments at Booking.com — ELITE CAMP 2018 is the CXL talk on opening experimentation to a whole company.
At scale the problems stop being about one test. (Paper) Top Challenges from the first Practical Online Controlled Experiments Summit is the paper written by the summit's participants from Microsoft, Google, LinkedIn, Airbnb, Facebook, Lyft, Netflix and the other organizations its abstract lists:
Online controlled experiments (OCEs), also known as A/B tests, have become ubiquitous in evaluating the impact of changes made to software products and services. While the concept of online controlled experiments is simple, there are many practical challenges in running OCEs at scale. To understand the top practical challenges in running OCEs at scale and encourage further academic and industrial exploration, representatives with experience in large-scale experimentation from thirteen different organizations (Airbnb, Amazon, Booking.com, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, Yandex, and Stanford University) were invited to the first Practical Online Controlled Experiments Summit. All thirteen organizations sent representatives. Together these organizations have tested more than one hundred thousand experiment treatments last year. Thirty-four experts from these organizations participated in the summit in Sunnyvale, CA, USA on December 13-14, 2018. While there are papers from individual organizations on some of the challenges and pitfalls in running OCEs at scale, this is the first paper to provide the top challenges faced across the industry for running OCEs at scale and some common solutions
The same causal question is older than software. (book) Modern epidemiology is Modern Epidemiology, Fourth Edition, which provides a complete desk reference of the methods of modern epidemiology, with contributions from the 4 authors plus 35 leading experts from various subfields.
Back in software, the hard part is reading the results. Challenges, Best Practices and Pitfalls in Evaluating Results of Online Controlled Experiments is the tutorial on exactly that. A/B Testing is the gold standard to estimate the causal relationship between a change in a product and its impact on key outcome measures. It is widely used in the industry to test changes ranging from simple copy change or UI change to more complex changes like using machine learning models to personalize user experience. The key aspect of A/B testing is evaluation of experiment results. Designing the right set of metrics — correct outcome measures, data quality indicators, guardrails that prevent harm to business, and a comprehensive set of supporting metrics to understand the “why” behind the key movements is the #1 challenge practitioners face when trying to scale their experimentation program. On the technical side, improving sensitivity of experiment metrics is a hard problem and an active research area, with large practical implications as more and more small and medium size businesses are trying to adopt A/B testing and suffer from insufficient power. In this tutorial we will discuss challenges, best practices, and pitfalls in evaluating experiment results, focusing on both lessons learned and practical guidelines as well as open research questions.
The tutorial's slides are two PDFs:
Online Metrics Tutorial - Part I. Introduction
Online Metrics Tutorial - Part II. Best Practices
That tutorial's point about insufficient power is where CUPED returns in practice. Increasing experimentation accuracy and speed by using control variates is Etsy's account. In this article, we share details about our team’s journey to bring the statistical method known as CUPED to Etsy, and how it is now helping other teams make more informed product decisions, as well as shorten the duration of their experiments by up to 20%. We offer some perspectives on what makes such a method possible, what it took us to implement it at scale, and what lessons we have learned along the way.
Randomizing users is not always possible. (Microsoft) Why Tenant-Randomized A/B Test is Challenging and Tenant-Pairing May Not Work is Microsoft Research on tenant- or enterprise-randomized tests: why tenant-randomization is challenging, the solutions available, and why tenant-pairing may not work.
The clearest single CUPED walkthrough is (good) How to Double A/B Testing Speed with CUPED — Microsoft’s variance reduction that’s becoming industry standard. It opens with where the method came from and who adopted it:
Controlled-experiment Using Pre-Existing Data (CUPED) is a variance reduction technique created by Microsoft in 2013. Since then, it has been implemented at Netflix, Booking.com, BBC, and many others.
In short, CUPED uses pre-experiment data to control for natural variation in an experiment’s north star metric. By removing natural variation, we can run statistical tests that require a smaller sample size. CUPED can be added to virtually any A/B testing framework; it’s computationally efficient and fairly straightforward to code.
The same notes are in SAMPLE SELECTION.
The companies named in that quote each wrote up their own version. (Netflix) Improving the Sensitivity of Online Controlled Experiments: Case Studies at Netflix treats sensitivity as a controlled experiment's ability to detect differences in business metrics due to product changes. (Booking) How Booking.com increases the power of online experiments with CUPED is Booking's version. A comment on it adds the caveat: yes, CUPED could be used like regression methods to adjust for confounders, but random assignment lets you make reasonably confident claims about causality, and observational data cannot give the same confidence. (BBC) Increasing experimental power with variance reduction at the BBC is Frank Hopkins' article; it discusses how the Experimentation team have been accounting for pre-experiment variance in order to increase the statistical power of their experiments. (TripAdvisor) (good) Reducing A/B test measurement variance by 30%+ is the tripadvisor.com engineering write-up of the same goal.
Variance reduction still ends in a statistical test, so the practical guides come next. (medium) practitioner guide to is the Practitioner’s Guide to Statistical Tests. From Hubspot, How to Do A/B Testing: 15 Steps for the Perfect Split Test is for anyone who wants to discover what marketing really works for their audience, and Sample size calculation shows how to calculate the ideal sample size so results are statistically significant and the time frame is accurate. The same guide, filed as Statistical tests, is the one to return to when choosing the test. Vidhya on the topic has two pieces: A/B Testing Measurement Frameworks — Every Data Scientist Should Know is Mainak's article on the measurement frameworks behind an A/B test; A/B Testing for Data Science using Python – A Must-Read Guide for Data Scientists is Shipra Saxena on A/B Testing as a way to test your products and how data scientists leverage it. (Yandex) Online Evaluation for Effective Web Service Development is a tutorial at the conference KDD 2018, for engineers, designers, analysts, and managers who want web service development to be effectively data-driven.
## A/B Testing Tools
Once the method and the CUPED reading are in place, the remaining question is what runs the test. The R-shiny Tool is Frank Hopkins' CUPED R-Shiny Tool, built for controlling the variance that exists within an experimental metric using a strongly correlated pre-experiment metric (covariate). For a full platform, Wasabi open source platform is Lucian Lita's Meet Wasabi, an Open Source A/B Testing Platform on the Intuit Blog, a real-time, enterprise-grade A/B Testing platform battle-tested in FinTech products. Its Git repo, intuit/wasabi, now says the Wasabi A/B Testing service is an open source project that is no longer under active development or being supported. On the commercial side, Optimizely sells everything marketing and digital teams need to create content fast, test and personalize, and prove business impact.
Deprecated links
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
Wasabi open source platform. This address no longer opens: https://www.intuit.com/blog/innovative-thinking/meet-wasabi-an-open-source-ab-testing-platform/
Last updated
Was this helpful?