> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/data/engineering-intro/data-engineering.md).
# Data Engineering
Data engineering is the job that makes data usable, and it is not the same job as writing product software. This page starts with what the role is, then Job versus Service, then the CAP theorem notes beside the CAP interview questions. The same notes are in [Data Engineering Questions & Training](/data/engineering-intro/data-engineering-questions.md).
The first thing to get right is that the role has its own shape. [Data engineers are not software developers - a comparison](https://betterprogramming.pub/data-engineering-is-not-software-engineering-af81eb8d3949) is the side-by-side of the two jobs. The role also splits further: [Different Types of data engineers](https://medium.com/coriers/different-types-of-data-engineering-teams-6a1056986d3) is Ben Rogojan on how the data engineer role morphed drastically in a decade, from a time when employers expected data scientists to both calculate eigenvectors and write MapReduce jobs for Hadoop. The teams it separates are data infra, data platform, data engineers, data governance, and BI & analytics.
## Job vs Service
Whichever team a data engineer sits on, the first design choice in an ML pipeline is whether a step runs as a job or as a service. The interview question puts it this way:
Explain the difference and the reason to choose using Job over Service and vice versa. Give an example for a project where you had to make this choice, in the context of ML pipelines and walk through your reasoning.
### [Cap theorem](https://medium.com/data-science/cap-theorem-and-distributed-database-management-systems-5c2be977950e)
A service that keeps state across machines runs into the CAP theorem. The heading links Syed Sadat Nazrul's "CAP Theorem and Distributed Database Management Systems", which starts from the shift away from scaling vertically, with more powerful machines, toward expanding horizontally with more machines doing the same task in parallel. The same notes are in [CAP Theorem](#cap-theorem).
Two sources push the theorem further. [Cap](https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/) is Confluent's "Turning the database inside-out with Apache Samza", from the company building the foundational platform for data in motion. [Cap is changing](https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed/) is InfoQ's "CAP Twelve Years Later: How the 'Rules' Have Changed". The figure below is the CAP triangle, copied from the original hosted image.
Cap theorem
## CAP Theorem
With the theorem stated, the interview questions check whether you can apply it. Each of the first six is answered in FullStack.Cafe's "15 CAP Theorem Interview Questions (ANSWERED) For System Design Interview":
1. [What Is CAP Theorem?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
2. [Can you 'get around' or 'beat' the CAP Theorem?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
3. [Name some types of Consistency patterns](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
4. [What Do You Mean By High Availability (HA)?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
5. [What are A and P in CAP and the difference between them?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
6. [What does the CAP Theorem actually say?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
7. How it affects real world application (latency is availability in real world)
[Another great resource for CAP questions](https://github.com/henryr/cap-faq) is henryr's The CAP FAQ on GitHub. The diagrams below illustrate the theorem.
CAP Theorem
CAP Theorem
CAP Theorem
CAP Theorem
CAP Theorem
## Deprecated links
{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}
* Cap theorem. This address no longer opens:
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://www.mlcompendium.com/data/engineering-intro/data-engineering.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
