DATA ENGINEERING · QUESTION 101
How do you measure and improve data quality?
On this page
Data quality is fitness for a specific use, commonly assessed through dimensions such as accuracy, completeness, validity, consistency, uniqueness, and freshness. I define critical expectations with data owners, measure them at pipeline boundaries, monitor trends and affected records, and connect failures to remediation. A high aggregate score can hide a broken field or a problem concentrated in one group.
LET'S MAKE IT SIMPLE
First, the idea in plain English
Data can be complete but wrong, current but duplicated, or valid in format but meaningless. Quality depends on what the data is used for. Teams need checks that reflect important decisions, plus a way to find the source and owner when a check fails.
A USEFUL COMPARISON
Picture it this way
A grocery delivery can arrive on time and still contain the wrong items. “Good delivery” has several dimensions: accuracy, completeness, condition, and timing. Data quality similarly needs more than one score.
FOLLOW THE JOURNEY
Measure data quality dimensions and route failures to owners
Show checks across validity, completeness, uniqueness, consistency, and freshness.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Repeated symptoms should lead to durable source fixes.
A dataset passes dimension-specific checks; failed records and metrics are reported to an owner with lineage to the upstream source.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Repeated symptoms should lead to durable source fixes.
UNDER THE HOOD
How it works, step by step
Define use and dimensions
Identify which fields and quality dimensions matter to consumers.
Why it matters: Quality is relative to a decision and cannot be reduced to one universal metric.
Measure at boundaries
Run checks for schema, range, uniqueness, relationships, and freshness.
Why it matters: Early validation localizes failures.
Report impact
Show affected rows, partitions, trends, and downstream dependencies.
Why it matters: Owners need actionable diagnostics rather than a red dashboard alone.
Remediate and prevent
Fix upstream causes, quarantine invalid data, and add regression checks.
Why it matters: Repeated symptoms should lead to durable source fixes.
OUTSIDE THE TEXTBOOK
A practical example
An order dataset checks unique order IDs, nonnegative quantities, valid currency codes, and arrival within an agreed freshness window. When one source begins sending duplicate IDs, the pipeline quarantines affected partitions and identifies the producing service rather than publishing inaccurate revenue totals.
GO A LITTLE DEEPER
Technical details that matter
- Accuracy may require trusted reference data or human review; syntactic validation alone cannot prove truth.
- Freshness and completeness metrics should account for expected schedules and seasonal volume.
- Lineage and ownership make quality alerts useful for root-cause response.
MAKE A GOOD CHOICE
When is this useful?
- Reliable reporting and financial aggregates.
- Model training where missing or invalid values affect decisions.
- Shared data products with explicit service expectations.
For exploratory analysis, label quality limitations clearly and avoid enforcing expensive gates that do not match the use.
WATCH OUT FOR
Common misunderstandings
- Using one aggregate quality score without field-level diagnostics.
- Treating valid schema as proof of accurate content.
- Alerting on anomalies without an owner or response plan.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Some dimensions are difficult to measure automatically.
- Strict checks can block legitimate new cases.
- Reference labels and rules may themselves be incomplete.
WHAT TO REMEMBER
Key takeaways
- 1Define quality for the intended use.
- 2Measure multiple dimensions at meaningful boundaries.
- 3Connect failures to records, lineage, owners, and remediation.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
What is the difference between validity and accuracy?
Validity checks whether data conforms to rules or formats; accuracy asks whether it reflects the real-world value correctly.
Should every anomaly block a pipeline?
No. Severity, downstream risk, and expected variation determine whether to block, quarantine, or warn.
SAY IT OUT LOUD
How to explain this in an interview
“I define data quality around its use, then check dimensions such as validity, completeness, uniqueness, consistency, and freshness. I want failures to identify affected records and upstream owners, not just generate an aggregate score. The right response may be to block, quarantine, or warn depending on risk, and recurring failures should lead to source fixes.”
Connect this question to the roadmap topic it supports.