Best practices for data quality in lakeflow

[gowri_databrick]

Hi everyone,

What are the recommended best practices for implementing data quality checks in Lake flow Spark Declarative Pipelines?

Should data quality expectations be applied mainly in the Bronze layer, Silver layer, or both?

Thanks!

[JamesBennett]

I’d use expectations in both Bronze and Silver, but for different purposes. Bronze should stay close to the raw source, so I’d use lightweight checks there to detect obvious ingestion/schema problems without aggressively cleaning the data. The heavier business and domain rules usually make more sense at the Bronze-to-Silver boundary, where invalid records can be dropped, quarantined, or cause the update to fail depending on their severity. Databricks also describes this boundary as a common place for quality gates.

For example, Bronze could check things like required fields and valid data types, while Silver could enforce business rules such as valid ranges, uniqueness, referential consistency, and domain-specific constraints. Lakeflow expectations support warn, drop, and fail behaviors, so you can choose the appropriate action per rule.

I’d also avoid duplicating every expectation across every layer. Keep the rules closest to where they provide the most value and use the pipeline event log to monitor failures and trends. That gives you a cleaner pipeline without sacrificing data quality. pbskidsgames.us.com can be a quick break after debugging the pipeline.

[balajij8]

@gowri_databrick You can adopt a tiered data quality strategy where the Silver layer handles the bulk of the DQ expectations enforcement while Bronze remains raw and unconstrained and Gold stays focused on reporting logic.

Bronze Layer - Raw Ingestion and Auditability
Keep Bronze as an immutable, append only zone. You can avoid filtering or dropping records in it. You can add the incoming data with technical metadata such as _ingested_at and _metadata.file_path for auditing and debugging. Apply expectations only if an issue would cause a pipeline-critical failure, using ON VIOLATION FAIL UPDATE for mandatory system level fields. Preserving raw data ensures you can always replay or reprocess historical records if downstream rules change.

Silver Layer Primary Data Quality Enforcement
You can do the primary data quality and cleansing as a centralized DQ. You can do type casting, standardization, deduplication and enforcing domain specific business rules. You can Use CONSTRAINT ON VIOLATION DROP ROW to filter out malformed or untrusted records and use ON VIOLATION FAIL UPDATE when critical business rules fails. Enforcing quality rules in Silver ensures all downstream consumers query clean, trusted and standardized datasets without duplicating validation logic across pipelines.

Gold Layer Aggregation and Business Metrics
You can keep minimal expectations in Gold layer as Silver already has cleansed data. Avoid re-validating individual row-level schemas here. You can restrict checks to aggregation specific constraints, dimensional integrity such as key uniqueness and reporting threshold validations before data is exposed to BI tools.

Bronze ensures total replayability and auditability, Silver provides a single source of truth for validation rules, Gold remains performant and simple. Data lineage makes it straightforward to trace exactly where and why records were filtered.

[data_pulse]

@gowri_databrick 
Recommended practice is to apply data quality checks in both Bronze and Silver with different purposes and levels of strictness.

• Bronze: apply lightweight structural checks such as schema validity, mandatory fields, corrupt records and ingestion metadata, while preserving raw data for replay.
• Silver: apply stronger business rules such as uniqueness, referential integrity, valid ranges, completeness, deduplication and reconciliation before downstream use.
• Use Lakeflow Declarative Pipeline expectations (DLT expectations) for native pipeline checks, or frameworks such as Databricks Labs DQX / Great Expectations / Soda for reusable, configuration-driven DQ rules.
• Use warn / drop / fail policies based on severity, and route invalid records to a quarantine table for investigation, remediation and possible reprocessing.
• Capture DQ metrics such as failed records, rule violations and pass rates, then expose them through Databricks dashboards / event logs / Log Analytics / alerts for operational monitoring.
• Gold can also have business level reconciliation checks, but the main quality gate is typically at the Silver layer. Typically Gold / Aggregate layer DQ Checks to ensure only business valid data is exposed downstream for reporting / file based extracts to another system.

Summary: Bronze validates ingestion quality, Silver enforces business quality, quarantine preserves rejected data, and DQ metrics provide ongoing observability and governance.

[srini_ve]

@gowri_databrick ,

In my experience, the best approach to data quality in DBR Lakeflow is to treat it as a continuous process rather than a one-time validation step.

A practical pattern is:

Bronze – Observe: Keep the raw data as close to the source as possible, while using quality checks primarily to identify and measure data quality issues.

Silver – Enforce: Apply most of the data quality rules here, including null checks, duplicate detection, data type validation, domain validation, and business key validation. Invalid records should ideally be quarantined rather than silently dropped.

Gold – Validate: Focus on business-level quality checks, reconciliations, aggregates, and KPI validation to ensure the data is fit for consumption.

Lakeflow expectations make this easier by allowing us to define how quality violations should be handled, whether to warn, drop the record, or fail the pipeline. The appropriate action should depend on the severity and business criticality of the rule.

To improve scalability and maintainability, I follow a metadata-driven approach. Rather than hardcoding data quality rules for each table, I centrally manage the rules, making them reusable, scalable, consistent, and easier to maintain.