Data Architecture

A platform needs an architecture before the warehouses and pipelines stack up. The page walks through a few data-platform architectures, from a modular stack to multi-petabyte and evolving platforms, and ends with an introduction to domain-driven design for data.

The architecture question and the ownership question are the same question seen from two sides. The same notes are in Data Mesh and ML Architecture.

The first architecture is the smallest one that still holds together: How to build a modular data stack is the modular data-platform write-up. Scale changes the picture. Nielsen shows a multi-petabyte data platform using EMR, the AWS re:Invent 2020 talk on how they built it on Amazon EMR. A platform is also never finished, and Evolving and scaling the data platform at Yotpo is the podcast episode that treats building a data platform as an iterative, evolutionary process that needs collaboration. Once the platform grows past one team, the boundaries have to follow the business, which is where the Intro to domain-driven design (DDD) comes in, the first part of a two-part introduction.