> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/data/engineering-intro/data-lineage.md). # Data Lineage When a number is wrong, someone has to walk the path the data took. The page defines data lineage and why it matters, then points at a few good articles, and ends with the vendors that build lineage tools. Lineage is where governance and quality meet: you cannot own or trust a number you cannot trace. The same notes are in [Data Governance](/data/engineering-intro/data-governance.md) and [Data Quality](/data/engineering-intro/data-quality.md). Data lineage refers to the detailed history of data as it moves through various stages and transformations in an information system. It is essentially the life cycle of data, from its origins to its endpoint, including how it is modified and processed over time. Understanding data lineage is crucial for several reasons: * Traceability — it helps track where data comes from, which is vital for debugging issues, understanding dependencies, and ensuring data quality. * Compliance — many regulatory requirements, such as GDPR and HIPAA, require knowing the flow of data to ensure it is handled securely and within legal parameters. * Data Governance — it aids in managing data, understanding its utility, and ensuring that data usage is consistent with organizational policies. * Impact Analysis — it allows organizations to assess the potential impact of changes in the data environment. This is crucial for risk management and strategic planning. * Audit and Reporting — data lineage provides transparency for audits, ensuring that all data used in financial reporting, for instance, is accurate and verifiable. Those reasons all depend on the trail actually existing. Tools and systems that manage data lineage collect metadata from various parts of data handling systems, providing a visual or documented trail of how data flows through software and systems, which transformations it undergoes, and how it is used in different analyses and decisions. This capability is particularly important in complex systems where data is handled across various platforms and services. ## Good articles The definition above is the short version; these three articles are the longer reads. The first two answer what data lineage is, from Octopai and Ardoq, and the third is Select Star's complete guide to its benefits, techniques, and best practices. ## Data lineage vendors Once the trail is understood, someone has to collect it, and that is what the lineage vendors sell. Many of them are also data catalogs. The same notes are in [Data Catalogs](/data/datasets-intro/data-catalogs.md). The cards below are Octopai, Collibra, Azure Purview, Cloudera, Alation, and the open-source Apache Atlas. Alation now presents its product as the Alation Intelligence Operating System, aimed at getting enterprise AI right and keeping it right. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://www.mlcompendium.com/data/engineering-intro/data-lineage.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.