Data Engineering
Data engineering is the job that makes data usable, and it is not the same job as writing product software. This page starts with what the role is, then Job versus Service, then the CAP theorem notes beside the CAP interview questions. The same notes are in Data Engineering Questions & Training.
The first thing to get right is that the role has its own shape. Data engineers are not software developers - a comparison is the side-by-side of the two jobs. The role also splits further: Different Types of data engineers is Ben Rogojan on how the data engineer role morphed drastically in a decade, from a time when employers expected data scientists to both calculate eigenvectors and write MapReduce jobs for Hadoop. The teams it separates are data infra, data platform, data engineers, data governance, and BI & analytics.
Job vs Service
Whichever team a data engineer sits on, the first design choice in an ML pipeline is whether a step runs as a job or as a service. The interview question puts it this way:
Explain the difference and the reason to choose using Job over Service and vice versa. Give an example for a project where you had to make this choice, in the context of ML pipelines and walk through your reasoning.
Cap theorem
A service that keeps state across machines runs into the CAP theorem. The heading links Syed Sadat Nazrul's "CAP Theorem and Distributed Database Management Systems", which starts from the shift away from scaling vertically, with more powerful machines, toward expanding horizontally with more machines doing the same task in parallel. The same notes are in CAP Theorem.
Two sources push the theorem further. Cap is Confluent's "Turning the database inside-out with Apache Samza", from the company building the foundational platform for data in motion. Cap is changing is InfoQ's "CAP Twelve Years Later: How the 'Rules' Have Changed". The figure below is the CAP triangle, copied from the original hosted image.
CAP Theorem
With the theorem stated, the interview questions check whether you can apply it. Each of the first six is answered in FullStack.Cafe's "15 CAP Theorem Interview Questions (ANSWERED) For System Design Interview":
How it affects real world application (latency is availability in real world)
Another great resource for CAP questions is henryr's The CAP FAQ on GitHub. The diagrams below illustrate the theorem.
Deprecated links
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
Cap theorem. This address no longer opens: https://towardsdatascience.com/cap-theorem-and-distributed-database-management-systems-5c2be977950e
Last updated
Was this helpful?