Best practices for data quality in lakeflow

[gowri_databrick]

Hi everyone,

What are the recommended best practices for implementing data quality checks in Lake flow Spark Declarative Pipelines?

Should data quality expectations be applied mainly in the Bronze layer, Silver layer, or both?

Thanks!

[JamesBennett]

I’d use expectations in both Bronze and Silver, but for different purposes. Bronze should stay close to the raw source, so I’d use lightweight checks there to detect obvious ingestion/schema problems without aggressively cleaning the data. The heavier business and domain rules usually make more sense at the Bronze-to-Silver boundary, where invalid records can be dropped, quarantined, or cause the update to fail depending on their severity. Databricks also describes this boundary as a common place for quality gates.

For example, Bronze could check things like required fields and valid data types, while Silver could enforce business rules such as valid ranges, uniqueness, referential consistency, and domain-specific constraints. Lakeflow expectations support warn, drop, and fail behaviors, so you can choose the appropriate action per rule.

I’d also avoid duplicating every expectation across every layer. Keep the rules closest to where they provide the most value and use the pipeline event log to monitor failures and trends. That gives you a cleaner pipeline without sacrificing data quality. pbskidsgames.us.com can be a quick break after debugging the pipeline.

[balajij8]

@gowri_databrick You can adopt a tiered data quality strategy where the Silver layer handles the bulk of the DQ expectations enforcement while Bronze remains raw and unconstrained and Gold stays focused on reporting logic.

Bronze Layer - Raw Ingestion and Auditability
Keep Bronze as an immutable, append only zone. You can avoid filtering or dropping records in it. You can add the incoming data with technical metadata such as _ingested_at and _metadata.file_path for auditing and debugging. Apply expectations only if an issue would cause a pipeline-critical failure, using ON VIOLATION FAIL UPDATE for mandatory system level fields. Preserving raw data ensures you can always replay or reprocess historical records if downstream rules change.

Silver Layer Primary Data Quality Enforcement
You can do the primary data quality and cleansing as a centralized DQ. You can do type casting, standardization, deduplication and enforcing domain specific business rules. You can Use CONSTRAINT ON VIOLATION DROP ROW to filter out malformed or untrusted records and use ON VIOLATION FAIL UPDATE when critical business rules fails. Enforcing quality rules in Silver ensures all downstream consumers query clean, trusted and standardized datasets without duplicating validation logic across pipelines.

Gold Layer Aggregation and Business Metrics
You can keep minimal expectations in Gold layer as Silver already has cleansed data. Avoid re-validating individual row-level schemas here. You can restrict checks to aggregation specific constraints, dimensional integrity such as key uniqueness and reporting threshold validations before data is exposed to BI tools.

Bronze ensures total replayability and auditability, Silver provides a single source of truth for validation rules, Gold remains performant and simple. Data lineage makes it straightforward to trace exactly where and why records were filtered.