Active Learning
When labels are expensive, the model should choose which examples a human labels next. The page starts with tutorials and surveys, then the warnings and pitfalls around query strategies such as query-by-committee, then tools and videos, and ends with Robert Munro's human-in-the-loop book and its uncertainty and diversity sampling cheat sheets.
The same notes are in Active Learning Algorithms and Annotation & Disagreement.
If you need to start somewhere start here - types of AL, the methodology, examples, sample selection functions. After that tutorial, the reference is Burr Settles' "Active Learning Literature Survey", Computer Sciences Technical Report 1648 at the University of Wisconsin–Madison, updated on January 26, 2010. It is a thorough thorough about review paper, and in practice The book on AL.
The survey promises cheaper labels, and the first warning is that the promise is conditional. Choose your model first, then do AL, from lighttag is LightTag's "Active Learning: Optimization Is Not Improvement": optimizing model uncertainty does not necessarily improve the dataset or the business outcome. The alternative is Query by committee - Importantly, the active learning method we presented above is the most naive form of what is called "uncertainty sampling" where we chose to sample based on how uncertain our model was. An alternative approach, called Query by Committee, maintains a collection of models (the committee) and selecting the most "controversial" data point to label next, that is one where the models disagreed on. Using such a committee may allow us to overcome the restricted hypothesis a single model can express, though at the onset of a task we still have no way of knowing what hypothesis we should be using.
The same notes are in Ensembles and Entropy / Information Gain.
The same model dependence shows up in the papers. This Paper: warning against transferring actively sampled datasets to other models. How to increase accuracy with AL starts from the point that more labeled samples give more success, but labels are costly and slow to obtain, which is what active learning algorithms emerged to overcome. AL with model selection - paper, from Carnegie Mellon's Machine Learning Department. Using weak and strong oracle in AL is the Chalmers University of Technology paper from the Department of Signals and Systems.
Those papers each fix one choice; the pitfalls paper lists all of them. The pitfalls of AL - how to choose (cost-effectively) the active learning technique when one starts without the labeled data needed for methods like cross-validation; 2. how to choose (cost-effectively) the base learning technique when one starts without the labeled data needed for methods like cross-validation, given that we know that learning curves cross, and given possible interactions between active learning technique and base learner; 3. how to deal with highly skewed class distributions, where active learning strategies find few (or no) instances of rare classes; 4. how to deal with concepts including very small subconcepts (“disjuncts”)—which are hard enough to find with random sampling (because of their rarity), but active learning strategies can actually avoid finding them if they are misclassified strongly to begin with; 5. how best to address the cold-start problem, and especially 6. whether and what alternatives exist for using human resources to improve learning, that may be more cost efficient than using humans simply for labeling selected cases, such as guided learning [3], active dual supervision [2], guided feature labeling [1], etc.
Knowing when to stop is its own question: the Confidence based stopping criteria paper is "Confidence-Based Stopping Criteria for Active Learning for Data Annotation" by Jingbo Zhu and Huizhen Wang (Northeastern University), Eduard Hovy (University of Southern California), and Matthew Ma (Scientific Works). The setting behind all of this is that a lot of unlabeled data is plentiful and cheap, eg. documents off the web, speech samples, images and video, as a great tutorial puts it.
From papers to working systems: AWS Sagemaker Active Learning, using annotation consolidation that finds outliers and weights accordingly, then takes that data, trains a model with the annotation + training data, if labeled with high probability, will use those labels, otherwise will re-annotate. For a first look on video, there is An ok video. For code, Active learning framework in python is bwallace's curious_snake. Active Learning Using Pre-clustering clusters before it queries. For language, A literature survey of active machine learning in the context of natural language processing is the NLP survey. Practical Online Active Learning for Classification is the online version, linked at the end of the body. Video 2 is a second video. Active learning in R - code is an exercise in active learning implementation in R on the MNIST dataset. Deep bayesian active learning with image data takes on the difficulty that AL methods rely on learning and updating models from small amounts of data, which deep learning tools do not do well.
Active learning puts a human back in the loop. Integrating Human-in-the-Loop (HITL) in machine learning is a necessity, not a choice. Here’s why? is the case for it, by Andy Kelly, and the figure below is its basic framework for HITL, credited to Supriya Ghosh.
Human In The loop ML book by Robert munro
The HITL framework has a full book behind it: Robert (Munro) Monarch's Human-in-the-Loop Machine Learning, with related cheat sheets. GIT is the PyTorch library for active learning that accompanies the book. Active transfer learning is "Active Transfer Learning with PyTorch": models adapted to predict their own errors, combining active learning and transfer learning.
The first cheat sheet is about confidence. When a Supervised Machine Learning model makes a prediction, it often gives a confidence in that prediction; if the model is uncertain, human feedback can help, and that is Uncertainty sampling. Its four measures are:
Least Confidence: difference between the most confident prediction and 100% confidence
Margin of Confidence: difference between the top two most confident predictions
Ratio of Confidence: ratio between the top two most confident predictions
Entropy: difference between all predictions, as defined by information theory
Uncertainty alone finds what confuses the model, not what is missing from the data. Diversity sampling - you want to make sure that it covers as diverse a set of data and real-world demographics as possible. Its four strategies are:
Model-based Outliers: sampling for low activation in your logits and hidden layers to find items that are confusing to your model because of lack of information
Cluster-based Sampling: using Unsupervised Machine Learning to sample data from all the meaningful trends in your data’s feature-space
Representative Sampling: sampling items that are the most representative of the target domain for your model, relative to your current training data
Real-world diversity: using sampling strategies that increase fairness when trying to support real-world diversity
The two are strongest together. Combine uncertainty sampling and diversity sampling is the "Advanced Active Learning Cheatsheet", on selecting the optimal unlabeled data for a human to review when systems are trained on thousands or millions of human-labeled examples. Its ten combinations:
Least Confidence Sampling with Clustering-based Sampling: sample items that are confusing to your model and then cluster those items to ensure a diverse sample (see diagram below).
Uncertainty Sampling with Model-based Outliers: sample items that are confusing to your model and within those find items with low activation in the model.
Uncertainty Sampling with Model-based Outliers and Clustering: combine methods 1 and 2.
Representative Cluster-based Sampling: cluster your data to capture multinodal distributions and sample items that are most like your target domain (see diagram below).
Sampling from the Highest Entropy Cluster: cluster your unlabeled data and find the cluster with the highest average confusion for your model.
Uncertainty Sampling and Representative Sampling: sample items that are both confusing to your current model and the most like your target domain.
Model-based Outliers and Representative Sampling: sample items that have low activation in your model but are relatively common in your target domain.
Clustering with itself for hierarchical clusters: recursively cluster to maximize the diversity.
Sampling from the Highest Entropy Cluster with Margin of Confidence Sampling: find the cluster with the most confusion and then sample for the maximum pairwise label confusion within that cluster.
Combining Ensemble Methods and Dropouts with individual strategies: aggregate results that come from multiple models or multiple predictions from one model via Monte-Carlo Dropouts aka Bayesian Deep Learning.
Active transfer learning, from the PyTorch article above, is the next diagram.
Machine in the loop flips the roles: the model suggests and the human confirms. LightTag's "When Should You Use Machine in the Loop?" explains that machine-in-the-loop labeling can reduce annotation cost while introducing bias, and how to reason about the tradeoff: Similar to AL, just a machine / model / algo adds suggestions. This is obviously a tradeoff of bias and clean dataset
Practical Online Active Learning for Classification, from the Department of Computer Science and Engineering at the University of California, San Diego, is http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.87.5536&rep=rep1&type=pdf
Deprecated links
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
AL with model selection - paper. This address no longer opens: http://www.alnurali.com/papers/paper_aaai_2014.pdf
A literature survey of active machine learning in the context of natural language processing. This address no longer opens: http://eprints.sics.se/3600/
Mnist competition (unpublished) using AL. This address no longer opens: http://dag.cvc.uab.es/mnist/statistics/
Medium on AL. This address no longer opens: https://news.voyage.auto/active-learning-and-why-not-all-data-is-created-equal-8a43a758c6f9
Uncertainty sampling. This address no longer opens: https://towardsdatascience.com/uncertainty-sampling-cheatsheet-ec57bc067c0b
Diversity sampling - you want to make sure that it covers as diverse a set of data and real-world demographics as possible. This address no longer opens: https://towardsdatascience.com/https-towardsdatascience-com-diversity-sampling-cheatsheet-32619693c304
Combine uncertainty sampling and diversity sampling. This address no longer opens: https://towardsdatascience.com/advanced-active-learning-cheatsheet-d6710cba7667
Last updated
Was this helpful?