> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/data/engineering-intro/data-engineering-questions.md).

# Data Engineering Questions

Interview loops and training prompts need a bank of questions, not a second textbook. The page starts with the general prompts, then points at larger question banks, and ends with the references the answers rest on, from CAP and ACID to join strategies. The same notes are in [Data Engineering](/data/engineering-intro/data-engineering.md).

## General

The general prompts come first, because they test experience before they test theory. The first eight come from LinkedIn's interviewing resources for data engineers, which cover interview strategies, the best questions for assessing a candidate, and how to improve the interview process; the rest are about BigQuery, Spark, and ETL.

1. [What scales of data have you worked with in the past?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
2. [How do you generally work with the departments that make use of your data?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
3. [Tell me about a time you had performance issues with an ETL. How did you identify this as a performance issue and how did you fix it?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
4. [Describe a time when you found a new use case for an existing database.](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
5. [Describe the most challenging project you’ve worked on. What was your role?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
6. [Think back to a project you’re proud of. What was it that gave you that sense of pride and accomplishment?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
7. [What do you consider to be one of the biggest mistakes you’ve ever made in your previous job?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
8. [Explain the differences between stream processing and data processing, with one caveat: pretend that I’m not familiar with data at all.](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
9. What are the considerations when choosing methods of ingesting data to BigQuery?
10. Have you worked with data science teams? What were your responsibilities?
11. What are the considerations of choosing Spark vs BigQuery?
12. What are the differences between ETL and ELT?

For the last question, [1](https://www.guru99.com/etl-vs-elt.html) is Guru99's ETL vs ELT comparison, and [2](https://www.xplenty.com/blog/etl-vs-elt/) is Mark Smallcombe's five critical differences, weighing cost, data type and volume, and other factors that shape the integration method. A third source used to sit here and is kept at the end of the page.

The same notes are in [Data Pipelines](/data/processing-intro/data-pipelines.md).

The three figures below draw the difference; the first is credited to guru99/david taylor and the third to mark smallcombe.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-17c1971fdb2cb14105380dbeab4a247ab6d8f610%2F0?alt=media" alt=""><figcaption><p>ETL and ELT</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-3a073f02d3c78f0411922895049f94c4715f1d3e%2F1?alt=media" alt=""><figcaption><p>ETL and ELT</p></figcaption></figure>

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-24fd0fb1f88da56640cdf7840ff068381183a379%2F2?alt=media" alt=""><figcaption><p>ETL and ELT</p></figcaption></figure>

## More question banks

A dozen prompts is enough for one interview; training needs the larger banks.

The biggest is [More than 2000 questions for data engineers](https://github.com/OBenner/data-engineering-interview-questions), a GitHub repo of more than 2000 data engineer interview questions, shown in the figure below.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-e5c1aa9520ebc9f3138c012f96c9f3214afd341a%2F11?alt=media" alt=""><figcaption><p>More than 2000 questions for data engineers</p></figcaption></figure>

[More data engineering questions](https://realpython.com/data-engineer-interview-questions-python/) is Real Python's set of data engineer interview questions answered with Python.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-9eb39d05b1836c2b187f6946affc219934435868%2F12?alt=media" alt=""><figcaption><p>More data engineering questions</p></figcaption></figure>

[Even more qs](https://www.softwaretestinghelp.com/data-engineer-interview-questions/) is a list of the most frequently asked data engineer interview questions and answers, for preparing for an upcoming interview.

<figure><img src="https://83674056-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Mgd48oS5_duTKOVE_Et%2Fuploads%2Fgit-blob-dd8770e3e8e6b48e9619cc1120f77dd932dd5958%2F13?alt=media" alt=""><figcaption><p>Even more qs</p></figcaption></figure>

## References

Questions need answers, and the references are where the answers come from.

[DS leads](https://docs.google.com/document/d/1gdfJce0p7jx0ptHJt3NE3NIhvzZCXJ5O0V-dvJse0HI/edit) is a data engineer interviews document that opens with the CAP theorem and BigQuery questions, such as the difference in the implementation between partitions and clustering in BQ. For system design, the primer that teaches how to design large-scale systems has both [System design interview q’s with solutions](https://github.com/donnemartin/system-design-primer#system-design-interview-questions-with-solutions) and a [Cap theorem](https://github.com/donnemartin/system-design-primer#system-design-interview-questions-with-solutions) section. [2](https://github.com/henryr/cap-faq) is the CAP FAQ on GitHub. The (which is great) [3](https://github.com/kislerdm/data-engineering-interviews) is the data engineering interviews Q\&A by and for the data community, which great isn't complete.

The database guarantees are a three-part series. [ACID](https://bardoloi.com/blog/2017/02/26/db-deep-dive/) starts from how an amazing feat of engineering becomes commonplace and taken for granted, and lays out the ACID guarantees. [CAP](https://bardoloi.com/blog/2017/03/06/cap-theorem/) picks up from the ACID guarantees of traditional RDBMSs like MS SQL Server, Oracle, and MySQL, and shows where they run into scale. [PACLEC](https://bardoloi.com/blog/2017/03/06/pacelc-theorem/) takes the CAP theorem as the framework under most modern distributed databases and adds latency, following Daniel Abadi's critique of how CAP had been applied.

The last question is why the field exists at all. [Why do we need Data engineering?](https://podcastaddict.com/episode/116229803) (podcast) is the Data Engineering Podcast episode on proven patterns for building successful data teams, where many stakeholders with competing goals and many roles make data products hard to deliver.

The Spark join strategies article linked on the platforms page, Join strategies #2 from Towards Data Science, is also at its original address: <https://towardsdatascience.com/strategies-of-spark-join-c0e7b4572bcf>.

## Deprecated links

{% hint style="warning" %}
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
{% endhint %}

* 3. This address no longer opens: <https://blog.panoply.io/etl-vs-elt-the-difference-is-in-the-how>
* Can you explain the parquet file format? This address no longer opens: <https://parquet.apache.org/documentation/latest/>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://www.mlcompendium.com/data/engineering-intro/data-engineering-questions.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.