Data Contract

Producers and consumers need a promise about the shape and meaning of a table. The page first defines data contracts and collects the articles that argue for them, then turns to the implementation options, from JSON Schema to protobuf and a schema registry.

A contract is what turns a table into a product and what makes quality testable. The same notes are in Data Product and Data Quality.

The definition comes from service design. A data contract is a formal agreement between a service and a client that abstractly describes the data to be exchanged. That is, to communicate, the client and the service do not have to share the same types, only the same data contracts. — Microsoft

The argument for bringing that idea into data comes from a series by Chad Sanderson. The following is a set of articles around the topics of data product / contract etc. however the focus is about data contracts and expectations IMO, therefore I place it here. — by Chad Sanderson. It starts with the problem: the existential threat of data quality, and why the Modern Data Stack can't solve it. Then the missing discipline: the death of data modeling part 1, and why it must be resurrected in the Modern Data Stack. Then the people problem: data's collaboration problem, and why the Modern Data Stack can't scale without fixing it. The answer is the rise of data contracts, and why data pipelines don't scale without them. production grade data products is the production-grade data pipeline, and why tech debt accumulates without contracts. And (finally!) a guide to data contracts p1 is the engineer's guide, implementing data contracts for entities.

Beyond that series, the (good) contracts, robustness in datamesh (but not only) is by Piethein Strengholt — also has about data-sharing agreements. The (good) from zero to hero is good mehdio, pragmatic tips on how to get started with data contracts after all the social-media noise about what they are. data contract experimentation is a SwirlAI newsletter issue covering data contracts alongside Spark shuffle, experimentation environments, and real-time model observability.

Implementation

A promise needs a format both sides can check, so the next step is choosing how to write the contract down.

The simplest format is JSON Schema, a vocabulary that allows you to annotate and validate JSON documents; the example page shows it in practice.

The heavier option is protobuf & gRPC. The source is Protobuf on git, the Python part of the repo for Google's data interchange format. google dev is the Protocol Buffers home page, describing them as language-neutral, platform-neutral extensible mechanisms for serializing structured data. what is gRPC? is the introduction to gRPC and protocol buffers, and what are proto buffers, what do they solve and what are the benefits? is the protobuf overview that answers those questions. For hands-on work, gRPC — a basic pythonic tutorial is the basic introduction to gRPC in Python, and Protobuf — a pythonic tutorial is the Python programmer's introduction to working with protocol buffers.

Community write-ups fill in the why. (good) Intro to gRPC & Protobuf by. Trevor Kendrick links his Medium page. (good) What are Protocol Buffers and why they are widely used? by Dineshchandgr is the second. Understanding protobufs is the third, by Daniel Padua Ferreira, and it opens with this summary:

Protocol Buffers (protobuf) is a method of serializing structured data which is particulary useful to communication between services or storing data. It was designed by Google early 2001 (but only publicly released in 2008) to be smaller and faster than XML. Protobuf messages are serialized into a binary wire format which is very compact and boosts performance.

One more explainer is protobuf what and why? by Swaminathan Muthuveerappan. Slightly off topic — how to choose between grpc, graphql, rest is the exhaustive guide to picking the right API technology among the three, a choice that shapes the technology roadmap for years.

Once there are many contracts, they need a home. Managing proto files and other schema types such as avro or json schema can be done in kafka's schema registry.

The same notes are in Kafka.

The two (good) articles from above also exist at their original addresses. (good) contracts, robustness in datamesh (but not only) is at https://towardsdatascience.com/data-contracts-ensure-robustness-in-your-data-mesh-architecture-69a3c38f07db. Data Contracts - From Zero To Hero, (good) from zero to hero, a pragmatic approach to data contracts, is at https://towardsdatascience.com/data-contracts-from-zero-to-hero-343717ac4d5e.