Data Platforms
A data platform is where compute and storage meet so teams can run jobs without reinventing the lake each time. The page starts with Databricks as the platform idea in practice, then moves to the query engines and their costs in Athena, Spark, and BigQuery, compares Glue DataBrew with Data Wrangler, and closes with a GCP disk note. The same notes are in Data Product and ML Architecture.
Databricks
The first platform is Databricks, and the story there runs from the guarantee it makes (ACID), through its courses and core concepts, into the Spark APIs, optimizations, and best practices underneath.
The same notes are in Databricks.
A lake only behaves like a database if writes are safe, so the first question is what ACID transactions are. Databricks is ACID is the Databricks glossary entry that answers it, and the figure below is the same point.
With the guarantee in place, the DB Learning Library is where to learn the platform itself. The Free courses are the free slice of the Databricks training catalog, expert-led courses in data science, machine learning, and big data analytics. The Docs Optimization recommendations are the documentation on optimizations and performance recommendations on Databricks, and the Comprehensive Guide to Optimize Databricks, Spark and Delta Lake Workloads collects best practices and strategies for making data workloads faster and more efficient. Two starter courses split the platform by role: DB for ML tours the workspace and notebooks, then exploratory data analysis, feature engineering, and tracking and managing models with MLflow, while DB for Data Engineering covers the foundations of a basic data engineering workflow, the workspace, Unity Catalog, and the day-to-day building blocks, through paired demos and labs.
Outside the official library, the (good) Introduction & Tutorial is a good walk through cluster, notebook, table, SQL, DataFrame, and connections. The must know 7 concepts are DataCamp's tutorial on the seven must-know Databricks concepts for any data specialist, and the two figures below come from it.
Under the notebooks sits Spark, and the first API choice is RDD vs Dataframe vs Dataset. The 2016 official blog post is the tale of three Apache Spark APIs: their performance, their optimization benefits, and when to use each. The linkedin blog post goes over the differences among the three again, the comparison on youtube is the video version of the same comparison, and RDDs vs. Dataframes vs. Datasets – What is the Difference and Why Should Data Engineers Care? starts from Apache Spark being the first choice for data engineers and explains why the difference matters to them.
Once the API is chosen, the next concern is optimizations. Optimization recommendations on Databricks is the same documentation page on performance recommendations named above, and the Comprehensive Guide to Optimize Databricks, Spark and Delta Lake Workloads is the same best-practices guide, here read for tuning rather than learning. Why and How: Partitioning in Databricks is Eduard Popa on the single most common lever, partitioning.
Best practices close the Databricks story: the official docs give the best practices and recommendations for using Delta Lake on Databricks.
Athena
After Databricks, Athena is the serverless query engine where the questions turn to engine, cost, and performance tuning. The list below is a question checklist; the tuning item links the AWS post with its top-10 tips.
What is the engine behind Athena
How is Presto different from Spark? How does it affect your query planning?
Performance tuning — Top 10: partitioning, bucketing, compression, optimize file sizes, optimize columnar data store generation, query tuning, optimize order by, optimize group by, use approx functions, column selection. What are the tradeoffs (time vs cost)?
What is the cost composed of?
How can you calculate cost?
How can you optimize your queries (partitions, join order, limit tricks, etc)
What options do you have to limit the cost of Athena?
When would you use Athena vs Spark
Spark
The last Athena question, when to use Athena vs Spark, leads straight into Spark itself: join strategies, DataFrame vs Dataset, and adaptive query execution.
The same notes are in Spark.
The articles by sivaprasad mandapati are a good pool here: Several spark articles that can be used as candidate questions is his Medium page, and several of the notes below come from it.
Joins are where most Spark jobs spend their time. Join strategies #1 is Prabhath Vemula's best practices and optimization techniques from joining two big tables in Spark SQL, and Join strategies #2 walks the join strategies Spark uses internally, which helps with tricky joins and out-of-memory root causes. The questions to ask of both are how? Pros and cons. (broadcast hash, shuffle hash, shuffle sort merge, cartesian). A related question is what’s the difference between a data frame and a dataset?
The choice between the two most common strategies is Sort merge vs broadcast, a tuning article that starts from partitions spread across executors as Spark's bread and butter. Its finding is that a broadcast join is 4 times faster if one of the table is small and enough to fit in memory. Is broadcasting always a good solution? Absolutely no. If you are joining two data sets both are very large broadcasting any table would kill your spark cluster and fails your job.
When neither side is small, the lever is the shuffle, which is the subject of Shuffle & AQE, the second part of that tuning series. Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan. It does three things:
Dynamically coalescing shuffle partitions
Dynamically switching join strategies
Dynamically optimizing skew joins
BigQuery
After Spark, BigQuery brings back the Athena kind of question on another cloud: cost, partitions, clustering, and access. The checklist is:
What is the difference in the implementation between partitions and clustering in BQ?
What ways do you know to reduce query cost in BigQuery?
What is the BigQuery cost composed of? How can you reduce storage cost?
Did you ever encounter a memory error when running BigQuery? Why does it happen and how is it related to the Dremel implementations
How can you control the access to sensitive data in BigQuery?
What options do you have to limit the cost of BigQuery?
When using BigQuery ML to train TF models — what happens in the background?
AWS Glue DataBrew vs Data Wrangler
Beside the query engines, data still has to be prepared before training, and AWS offers two tools for it. Julien simon on AWS glue data brew vs data wrangler asks which one to use for data preparation, AWS Glue DataBrew or Amazon SageMaker Data Wrangler.
GCP
Closing the cloud shelf is a practical GCP problem: running out of disk mid-download or mid-training. Resize google disk size shows how to resize a persistent disk on the fly instead of stopping the VM and attaching a new disk. The references 1, and 2 back it up: the first is the Google Cloud documentation on creating and attaching persistent disk volumes on Windows and Linux VMs, and the second resizes a Compute Engine persistent disk without downtime or restarting the instance.
Deprecated links
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
Vector Search. This address no longer opens: https://www.databricks.com/training/catalog/new-capability-overview-vector-search-2535
How I Use Caching in Databricks to Increase Performance and Save Costs. This address no longer opens: https://blog.det.life/caching-in-databricks-explained-68c07bf1f76b
Last updated
Was this helpful?