cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Lakeflow Expectations: Dropping Bad Rows Can Hide Real Sales

amitsharma1707
Databricks Partner

 

A practical data quality lesson: choose the response to a failed rule based on its business impact.

Consider a hypothetical retailer using Azure Databricks. An incoming order has a valid order ID and sales amount, but its customer ID is missing.

A rule such as customer_id IS NOT NULL catches the problem. However, dropping that row from the reporting dataset also removes its sales amount. The remaining records pass validation, while reported revenue becomes incomplete.

Lakeflow pipeline expectations offer three responses:

  • Warn: Keep invalid records and collect quality metrics.

  • Drop: Exclude invalid records from the target and track the dropped count.

  • Fail: Stop the affected flow when a record violates the rule.

For this fictional scenario, I would preserve the original order, explicitly route unresolved records to a quarantine dataset, and define how corrected records return to processing. A drop expectation does not automatically create that recovery path.

I would also reconcile sales amounts across accepted and quarantined records before publishing totals. Passing row-level checks alone does not establish that a report is complete.

How does your team decide when a failed quality rule should retain a sales record, quarantine it, or block processing?

Reference: Azure Databricks pipeline expectations.

#Databricks #Lakeflow #DataQuality

Amit Sharma
0 REPLIES 0