> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/data/engineering-intro.md). # Data Engineering The dataset has to live in a system other people can run. After this chapter the reader can place SQL, storage, quality, lineage, and ownership on one path from source to table. The path starts with asking and storing. A warehouse only answers if you can ask it in SQL, so [SQL](/data/engineering-intro/sql.md) comes first: command groups, keys, indexes, and how sharding differs from partitioning. [Tools](/data/engineering-intro/tools.md) is the shelf for change capture, lakes, pipelines, BI, integration, and quality checks, opened by the job each group is for. Data then has to live somewhere queryable, and [Databases](/data/engineering-intro/lakes-and-warehouses.md) is the choice between lake, warehouse, table format, or a specialized engine. Because storage choices and models decide how a warehouse can be queried later, [Database Modeling](/data/engineering-intro/database-architecture-and-modeling.md) follows with warehouse modeling, Data Vault, data fabric, and virtualization. With storage in place, the chapter turns to the job and the system around it. [Data Engineering](/data/engineering-intro/data-engineering.md) is the job that makes data usable, which is not the same job as writing product software. A platform needs an architecture before the warehouses and pipelines stack up, and [Data Architecture](/data/engineering-intro/data-architecture.md) is the short list of data-platform architectures and domain-driven design for data. [Data Platforms](/data/engineering-intro/data-platforms.md) is where compute and storage meet so teams can run jobs without reinventing the lake each time. [Data Patterns](/data/engineering-intro/patterns.md) gives the rule for what history looks like when a row changes over time. Once tables flow, they have to be trusted. Bad rows break models and dashboards, so [Data Quality](/data/engineering-intro/data-quality.md) defines and tests quality instead of assuming it. Pipelines fail quietly unless someone watches freshness, volume, and schema drift, which is [Data Observability](/data/engineering-intro/data-observability.md). When a number is wrong, someone has to walk the path the data took, and that is [Data Lineage](/data/engineering-intro/data-lineage.md). Producers and consumers need a promise about the shape and meaning of a table, the [Data Contract](/data/engineering-intro/data-contract.md). Trust then becomes ownership. [Data Governance](/data/engineering-intro/data-governance.md) is about who may change the data and what "good" means across teams. A central data team can become the bottleneck, so [Data Mesh](/data/engineering-intro/data-mesh.md) treats domains as owners of data products, and [Data Product](/data/engineering-intro/data-product.md) is the work of treating data as something with users and owners, since a platform nobody adopts is not a product. The chapter ends with the organization that pays for all of it. [Data Strategy](/data/engineering-intro/data-strategy.md) says what data work is for before the company buys another tool. [Data Vision](/data/engineering-intro/data-vision.md) says what success looks like before the org chart is drawn. [Data Teams](/data/engineering-intro/data-teams.md) is the team shape that can ship analytics. [Data KPIs](/data/engineering-intro/data-kpis.md) measures the return on data spend, not only pipeline uptime. [Data Program Management](/data/engineering-intro/data-program-management.md) is the roles and definitions that keep projects from drifting. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://www.mlcompendium.com/data/engineering-intro.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.