> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/data/engineering-intro/data-observability.md).
# Data Observability
Pipelines fail quietly unless someone watches freshness, volume, and schema drift. Observability is that watching, done continuously instead of after a dashboard breaks. The page moves from the idea to the tools that do the watching, and then to what to do when an outage happens anyway. The same notes are in [Data Quality](/data/engineering-intro/data-quality.md).
## Tools
Watching data health by hand does not scale, so the idea turns into a set of tools and a cycle for acting on what they find.
[Monte Carlo](https://www.montecarlodata.com/) pitches itself as going beyond data quality to end-to-end data and AI observability for enterprise teams. [BigEye](https://www.bigeye.com/) is the data and AI trust platform for large enterprises, combining data observability, end-to-end lineage, and governance. [Databand](http://web.archive.org/web/20231208161414/https://databand.ai/), read through its archived page, is the proactive observability platform that tries to catch bad data before it impacts the business. Its explainer, [What is data observability](http://web.archive.org/web/20231208161417/https://databand.ai/data-observability/), defines the term as an umbrella of activities and technologies that let you identify, troubleshoot, and resolve data issues in near real-time, and it frames the work as a cycle:
> The DataOps cycle outlines the fundamental activities needed to improve how data is managed within the DataOps workflow. This cycle consists of three distinct stages: Detection, Awareness, and Iteration.
The figure shows those three stages as a loop.
The DataOps cycle
For an open-source option, [DQOps](https://dqops.com) is a data quality platform for the whole data platform lifecycle, from profiling new data sources to automating data quality monitoring.
## Outage handling
Detection only helps if the pipeline can recover once something breaks, and the architecture decides how painful that recovery is.
The note is about outage handling and the differences between stream-based processing vs concurrent isolated worker-based processing using
Q: you have a real time stream — what is better? A stream-based processing system, or a worker-based, that can be triggered on different time ranges, in the context of recovery from outage.
The figure below sketches that comparison; the credit is nielsen Ilai Malka.
Outage handling
## Deprecated links
{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}
* Databand. This address no longer opens:
* What is data observability and what is the dataOps cycle. This address no longer opens:
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://www.mlcompendium.com/data/engineering-intro/data-observability.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
.png?alt=media)