Data Quality

Bad rows break models and dashboards, so quality has to be defined and tested, not assumed. The page starts with what quality means and how to measure it, then turns to dbt as the tool that tests it, and ends with data validation notes and tools. The same notes are in Data Contract, Data Lineage, Data Observability, Data Testing, and Data Validation.

The first point is that quality is not a tool you buy. why cant data quality be fixed with tech is Sharon Rehana on why data quality matters, where bad data comes from, and how to fix those issues. Once the problem is owned, it has to be defined: CloverDX - data quality explains what data quality really means, how to measure it, and the common best practices and pitfalls.

Measuring means picking metrics. CloverDX - 6 Data Quality Metrics You Can't Afford To Ignore names six core data quality metrics, how they differ from dimensions and KPIs, and how to track them for reliable AI outcomes. FirstEigan - data quality metrics is Seth Rao's version of six key metrics, with examples and ways to automate them on Azure and Snowflake. Dataladder - data quality metrics widens the list to ten metrics you should measure. Datacademia - data quality metrics adds a warning about naming, that these metrics are often called dimensions although in a dimensional model they act as attributes of the data rules, and defines timeliness as the time between when information is expected and when it is readily available. At big-data scale quality becomes veracity, one of the V's: Open sistemas - big data challenges - veracity is the OpenSistemas overview of the Four V's of Big Data.

DBT

Once the metrics are named, they need to run as tests on every build, and dbt is where that happens in the warehouse.

The same notes are in Snowflake and Tools.

The starting point is the DBT builtin tests, the dbt docs on configuring data tests to assess the quality of input data and the accuracy of the resulting datasets. DBT expectations is the dbt package-hub entry that extends those tests. When neither fits, DBT custom generic tests shows how to define your own custom generic data tests. Those tests are written in Jinja, and Jinja is Real Python's introduction to Jinja templating.

Data Validation

dbt tests the warehouse; validation is the wider habit of guarding every place bad data can enter, and these notes come from the question bank.

The same notes are in Data Quality, Data Testing, and Tools.

The question is: how can you protect yourself from bad data? Data validation, TDDA, monitoring. Each answer has tools. Type validation: typeguard is a run-time type checker for Python. Data validation: pydantic validates Python dataclasses. Test driven: tdda is the test-driven data analysis functions library. Data quality: great expectations is GX Core, an open source framework for testing, validating, and documenting data quality across pipelines, workflows, and teams. Saas: SuperConductive by GE is the hosted side of the same Great Expectations platform.