ML Model Monitoring & Alerts
A model that ships without eyes goes blind. The page first covers why and how to watch models, dependencies, and features in production, then names drift, the change those alerts are for, and ends with the landscapes that compare monitoring tools.
Monitoring & alerts
Before drift has a name, the first job is to watch models, dependencies, features, and production performance at all.
The case for it is my own series. Monitor! Stop being a blind DS explains the many use cases and the monumental importance of monitoring and alerts from a data-science-researcher point of view, and reviews companies that try to solve this huge but strangely undiscussed problem. A model also breaks when what it depends on breaks, which is the point of Monitor your dependencies! Stop being a blind DS, with dominos as the analogy. The same argument aimed at leadership is Data science observability for executives.
Once you agree to watch, the question is what to watch. Production Machine Learning Monitoring: Outliers, Drift, Explainers & Statistical Performance starts from the line "The lifecycle of a machine learning model only begins once it’s in production". Alejandro Saucedo's talk on the same subject, Production Machine Learning Monitoring: Principles, Patterns and Techniques, is on youtube; it uses alibi-explain uses see compendium and Ali detect see compendium.
The logging side of the same problem is MLflow, HyperparameterHunter, Hyperopt, concept drift, unit tests Javier Rodriguez Zaurin. When there are many models, Yaron Gueta's Meta anomaly over multiple models, aggregate answers how to track machine learning algorithms with machine learning algorithms. What happens after your machine learning model is deployed? Vidhya on monitoring data & models is a framework for planning post-deployment monitoring. Features drift too: production feature distributions can move away from the baseline because the world changes, and Monitor ML features using Amazon SageMaker Feature Store and AWS Glue DataBrew is how to catch that when features are shared across teams.
The same notes are in Feature Stores & Feature Pipelines.
Drift
Once monitoring is on, drift is the change in data or labels that those alerts are for.
The same notes are in Anomaly Detection, Comparing distributions (distance methods), General patterns, Hyper Parameter Optimization, and Training Strategies.
Drift comes in two kinds, and the first two readings separate them. Philip Tannor's Data & concept drifts focuses on monitoring models once they are fully deployed and running in production. 2 is Explorium on understanding and handling data drift and concept drift. The hard case is when labels are missing: (good) Inferring Concept Drift Without Labeled Data. Also talks about stream-based drift by Cloudera — Fast Forward Labs.
Measuring drift is where Arize.ai's articles come in. Data, concept, feature drifts — various comparisons between train/prod/validation time windows, diff models, A/B testing etc., and how to measure drifts, using statistical distances on inputs, outputs, and actuals.
The same notes are in A/B Testing.
The same authors argue that teams rushing ML into production lack the right tools, and name the three they need: Model store, Feature store, evaluation store. Monitor model performance in production — real-time, biased, delayed, and no ground truth. The statistical-distance article also works as a list of use cases — i.e., how to use statistical differences/distances.
Detecting drift raises the question of what to do next. Some advice on Medium, relabel using latest model (can we even trust it?) retrain after. Uber argues that waiting is the mistake, in Adversarial Validation Approach to Concept Drift Problem in User Targeting Automation Systems at Uber — Previous research on concept drift mostly proposed model retraining after observing performance decreases. However, this approach is suboptimal because the system fixes the problem only after suffering from poor performance on new data. Here, we introduce an adversarial validation approach to concept drift problems in user targeting automation systems. With our approach, the system detects concept drift in new data before making inference, trains a model, and produces predictions adapted to the new data.
The same idea can be coded directly: a drift estimator between data sets using random forest; the formula is in the Medium article above; code here at MLBox. For a maintained library, Alibi-detect — is an open-source Python library focused on outlier, adversarial, and drift detection, by Seldon. And for why drift is missed in the first place, What is concept drift and why does it go undetected. Breaking down concept drit and explaining the best methods to avoid it.
The figure below, copied from the original hosted image, lists Alibi Detect's drift features.
Tool comparisons
With drift named, the remaining question is which tool does the watching, so these lists compare the monitoring landscapes and tool shelves.
The market map I built is at https://www.stateofmlops.com, the State of MLOps card view, and (by me) Medium is the MLOps Monitoring Market Review 2021 that goes with it. The same review has an article, open-source Airtable version on Airtable's low-code app platform. A curated list of MLOps projects that used to sit here now points to an unrelated site and is kept at the end of the page.
Other vendors and communities drew their own maps. Aporia kept one; the archived address now announces that Aporia has been acquired by Coralogix. Neptune.AI is the MLOPS Tools Landscape by Neptune. Ambiata wrote an MLOps tools post; the address now shows Ambiata's AI staff augmentation services. LakeFS on the state of data engineering — has monitoring and observability inside. The NLP Pandect, a comprehensive reference for all topics related to Natural Language Processing, has a section on MLOps for NLP. ml-ops.org is the community site, and Awesome production ML is the curated list of open source libraries to deploy, monitor, version, and scale machine learning, shown in the figure below.
Two of the articles above also live at their original addresses. Monitor ML features using Amazon SageMaker Feature Store and AWS Glue DataBrew. https://towardsdatascience.com/monitor-ml-features-using-amazon-sagemaker-feature-store-and-aws-glue-databrew-c530abcc479a. Concept drift is a drift of labels with time for the essentially the same data, by Ashok Chilakapati, and that is the some advice on medium article. https://towardsdatascience.com/concept-drift-and-model-decay-in-machine-learning-a98a809ea8d4
Monitoring does not stop at alerts. Modern distributed, event-driven, multi-cloud systems generate more signals and failure modes than human operators can manage in real time, so Self-Healing Agentic Systems (January 2026) is the monitoring note for a system that repairs itself, and the agent side is on the agents page.
For a concrete starting point, Monitor! Easy MLOps Model Monitoring With New Relic (April 2022) is a monitoring setup on this page.
Deprecated links
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
Production Machine Learning Monitoring: Outliers, Drift, Explainers & Statistical Performance. This address no longer opens: https://towardsdatascience.com/production-machine-learning-monitoring-outliers-drift-explainers-statistical-performance-d9b1d02ac158
Mlflow, Hyperparameterhunter,hyperopt, concept drift, unit tests. This address no longer opens: https://towardsdatascience.com/putting-ml-in-production-ii-logging-and-monitoring-algorithms-91f174044e4e
feature drifts. This address no longer opens: https://towardsdatascience.com/using-statistical-distance-metrics-for-machine-learning-observability-4c874cded78
Model store, Feature store, evaluation store. This address no longer opens: https://towardsdatascience.com/the-only-3-ml-tools-you-need-1aa750778d33
Monitor model performance in production. This address no longer opens: https://towardsdatascience.com/the-playbook-to-monitor-your-models-performance-in-production-ec06c1cc3245
use cases - i.e., how to use statistical differences/distances. This address no longer opens: https://towardsdatascience.com/using-statistical-distance-metrics-for-machine-learning-observability-4c874cded78
What is concept drift and why does it go undetected. Breaking down concept drit and explaining the best methods to avoid it. This address no longer opens: https://censius.ai/blogs/what-is-concept-drift-and-why-does-it-go-undetected
How does data drift hamper AI performance. Understand how data drift affect peak AI performance and how you can detect it. This address no longer opens: https://censius.ai/blogs/data-drift-barrier-to-ai-performance
by Aporia. This address no longer opens: https://aporia.com
Neptune.AI MLOPS tools landscape. This address no longer opens: https://mlops.neptune.ai/
Twimlai ML AI solutions. This address no longer opens: https://twimlai.com/solutions/
MLOps.toys. This address now points to an unrelated site: https://mlops.toys/
Last updated
Was this helpful?