Data Engineering Questions

Interview loops and training prompts need a bank of questions, not a second textbook. The page starts with the general prompts, then points at larger question banks, and ends with the references the answers rest on, from CAP and ACID to join strategies. The same notes are in Data Engineering.

General

The general prompts come first, because they test experience before they test theory. The first eight come from LinkedIn's interviewing resources for data engineers, which cover interview strategies, the best questions for assessing a candidate, and how to improve the interview process; the rest are about BigQuery, Spark, and ETL.

1. What scales of data have you worked with in the past?

2. How do you generally work with the departments that make use of your data?

3. Tell me about a time you had performance issues with an ETL. How did you identify this as a performance issue and how did you fix it?

4. Describe a time when you found a new use case for an existing database.

5. Describe the most challenging project you’ve worked on. What was your role?

6. Think back to a project you’re proud of. What was it that gave you that sense of pride and accomplishment?

7. What do you consider to be one of the biggest mistakes you’ve ever made in your previous job?

8. Explain the differences between stream processing and data processing, with one caveat: pretend that I’m not familiar with data at all.

9. What are the considerations when choosing methods of ingesting data to BigQuery?

10. Have you worked with data science teams? What were your responsibilities?

11. What are the considerations of choosing Spark vs BigQuery?

12. What are the differences between ETL and ELT?

For the last question, 1 is Guru99's ETL vs ELT comparison, and 2 is Mark Smallcombe's five critical differences, weighing cost, data type and volume, and other factors that shape the integration method. A third source used to sit here and is kept at the end of the page.

The same notes are in Data Pipelines.

The three figures below draw the difference; the first is credited to guru99/david taylor and the third to mark smallcombe.

ETL and ELT

ETL and ELT

ETL and ELT

More question banks

A dozen prompts is enough for one interview; training needs the larger banks.

The biggest is More than 2000 questions for data engineers, a GitHub repo of more than 2000 data engineer interview questions, shown in the figure below.

More than 2000 questions for data engineers

More data engineering questions is Real Python's set of data engineer interview questions answered with Python.

More data engineering questions

Even more qs is a list of the most frequently asked data engineer interview questions and answers, for preparing for an upcoming interview.

Even more qs

References

Questions need answers, and the references are where the answers come from.

DS leads is a data engineer interviews document that opens with the CAP theorem and BigQuery questions, such as the difference in the implementation between partitions and clustering in BQ. For system design, the primer that teaches how to design large-scale systems has both System design interview q’s with solutions and a Cap theorem section. 2 is the CAP FAQ on GitHub. The (which is great) 3 is the data engineering interviews Q&A by and for the data community, which great isn't complete.

The database guarantees are a three-part series. ACID starts from how an amazing feat of engineering becomes commonplace and taken for granted, and lays out the ACID guarantees. CAP picks up from the ACID guarantees of traditional RDBMSs like MS SQL Server, Oracle, and MySQL, and shows where they run into scale. PACLEC takes the CAP theorem as the framework under most modern distributed databases and adds latency, following Daniel Abadi's critique of how CAP had been applied.

The last question is why the field exists at all. Why do we need Data engineering? (podcast) is the Data Engineering Podcast episode on proven patterns for building successful data teams, where many stakeholders with competing goals and many roles make data products hard to deliver.

The Spark join strategies article linked on the platforms page, Join strategies #2 from Towards Data Science, is also at its original address: https://towardsdatascience.com/strategies-of-spark-join-c0e7b4572bcf.

Deprecated links

These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.

• 1. This address no longer opens: https://blog.panoply.io/etl-vs-elt-the-difference-is-in-the-how

• Can you explain the parquet file format? This address no longer opens: https://parquet.apache.org/documentation/latest/