> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/data/engineering-intro/data-contract.md). # Data Contract Producers and consumers need a promise about the shape and meaning of a table. The page first defines data contracts and collects the articles that argue for them, then turns to the implementation options, from JSON Schema to protobuf and a schema registry. A contract is what turns a table into a product and what makes quality testable. The same notes are in [Data Product](/data/engineering-intro/data-product.md) and [Data Quality](/data/engineering-intro/data-quality.md). The definition comes from service design. A data contract is **a formal agreement between a service and a client that abstractly describes the data to be exchanged**. That is, to communicate, the client and the service do not have to share the same types, only the same data contracts. — [Microsoft](https://learn.microsoft.com/en-us/dotnet/framework/wcf/feature-details/using-data-contracts) The argument for bringing that idea into data comes from a series by Chad Sanderson. The following is a set of articles around the topics of data product / contract etc. however the focus is about data contracts and expectations IMO, therefore I place it here. — by Chad Sanderson. It starts with the problem: [the existential threat of data quality](https://dataproducts.substack.com/p/the-existential-threat-of-data-quality), and why the Modern Data Stack can't solve it. Then the missing discipline: [the death of data modeling part 1](https://dataproducts.substack.com/p/the-death-of-data-modeling-pt-1), and why it must be resurrected in the Modern Data Stack. Then the people problem: [data's collaboration problem](https://dataproducts.substack.com/p/datas-collaboration-problem), and why the Modern Data Stack can't scale without fixing it. The answer is the rise of [data contracts](https://dataproducts.substack.com/p/the-rise-of-data-contracts), and why data pipelines don't scale without them. [production grade data products](https://dataproducts.substack.com/p/the-production-grade-data-pipeline) is the production-grade data pipeline, and why tech debt accumulates without contracts. And (finally!) [a guide to data contracts p1](https://dataproducts.substack.com/p/an-engineers-guide-to-data-contracts) is the engineer's guide, implementing data contracts for entities. Beyond that series, the (good) [contracts, robustness in datamesh (but not only)](https://medium.com/data-science/data-contracts-ensure-robustness-in-your-data-mesh-architecture-69a3c38f07db) is by Piethein Strengholt — also has about data-sharing agreements. The (good) [from zero to hero](https://medium.com/data-science/data-contracts-from-zero-to-hero-343717ac4d5e) is good mehdio, pragmatic tips on how to get started with data contracts after all the social-media noise about what they are. [data contract experimentation](https://www.newsletter.swirlai.com/p/sai-04-data-contracts-experimentation) is a SwirlAI newsletter issue covering data contracts alongside Spark shuffle, experimentation environments, and real-time model observability. ## Implementation A promise needs a format both sides can check, so the next step is choosing how to write the contract down. The simplest format is [**JSON Schema**](https://json-schema.org/), a vocabulary that allows you to **annotate** and **validate** JSON documents; the [example](https://json-schema.org/learn/miscellaneous-examples.html) page shows it in practice. The heavier option is protobuf & gRPC. The source is [Protobuf on git](https://github.com/protocolbuffers/protobuf/tree/main/python), the Python part of the repo for Google's data interchange format. [google dev](https://developers.google.com/protocol-buffers) is the Protocol Buffers home page, describing them as language-neutral, platform-neutral extensible mechanisms for serializing structured data. [what is gRPC?](https://grpc.io/docs/what-is-grpc/introduction/) is the introduction to gRPC and protocol buffers, and [what are proto buffers, what do they solve and what are the benefits?](https://developers.google.com/protocol-buffers/docs/overview) is the protobuf overview that answers those questions. For hands-on work, gRPC — [a basic pythonic tutorial](https://grpc.io/docs/languages/python/basics/) is the basic introduction to gRPC in Python, and Protobuf — [a pythonic tutorial](https://developers.google.com/protocol-buffers/docs/pythontutorial) is the Python programmer's introduction to working with protocol buffers. Community write-ups fill in the why. (good) Intro to gRPC & Protobuf by. [Trevor Kendrick](https://medium.com/@trevor.kendrick?source=post_page-----c21054ef579c--------------------------------) links his Medium page. (good) What are Protocol Buffers and why they are widely used? by [Dineshchandgr](https://medium.com/@dineshchandgr?source=post_page-----cbcb04d378b6--------------------------------) is the second. [Understanding protobufs](https://medium.com/danielpadua/understanding-protocol-buffers-protobuf-a466d8943df8) is the third, by [Daniel Padua Ferreira](https://medium.com/@danielpadua?source=post_page-----a466d8943df8--------------------------------), and it opens with this summary: > Protocol Buffers (protobuf) is a method of serializing structured data which is particulary useful to communication between services or storing data. It was designed by Google early 2001 (but only publicly released in 2008) to be smaller and faster than XML. Protobuf messages are serialized into a [binary wire](https://developers.google.com/protocol-buffers/docs/encoding) format which is very compact and boosts performance. One more explainer is protobuf what and why? by [Swaminathan Muthuveerappan](https://medium.com/@swamim?source=post_page-----fcb324a64564--------------------------------). Slightly off topic — [how to choose between grpc, graphql, rest](https://ashish-bania.medium.com/the-exhaustive-guide-to-choosing-between-grpc-graphql-and-rest-b7e4fd6d547e) is the exhaustive guide to picking the right API technology among the three, a choice that shapes the technology roadmap for years. Once there are many contracts, they need a home. Managing proto files and other schema types such as avro or json schema can be done in [kafka's schema registry](https://docs.confluent.io/platform/current/schema-registry/index.html). The same notes are in [Kafka](/ai-engineering/full-stack-and-ops-1/key-value-db.md#kafka). The two (good) articles from above also exist at their original addresses. (good) contracts, robustness in datamesh (but not only) is at . Data Contracts - From Zero To Hero, (good) from zero to hero, a pragmatic approach to data contracts, is at . --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://www.mlcompendium.com/data/engineering-intro/data-contract.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.