<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic Lakeflow Expectations: Dropping Bad Rows Can Hide Real Sales in Data Engineering</title>
    <link>https://community.databricks.com/t5/data-engineering/lakeflow-expectations-dropping-bad-rows-can-hide-real-sales/m-p/175627#M57025</link>
    <description>&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;A practical data quality lesson: choose the response to a failed rule based on its business impact.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Consider a hypothetical retailer using Azure Databricks. An incoming order has a valid order ID and sales amount, but its customer ID is missing.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;A rule such as &lt;/SPAN&gt;&lt;SPAN&gt;customer_id IS NOT NULL&lt;/SPAN&gt;&lt;SPAN&gt; catches the problem. However, dropping that row from the reporting dataset also removes its sales amount. The remaining records pass validation, while reported revenue becomes incomplete.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Lakeflow pipeline expectations offer three responses:&lt;/SPAN&gt;&lt;/P&gt;&lt;UL&gt;&lt;LI&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;Warn:&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; Keep invalid records and collect quality metrics.&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;LI&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;Drop:&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; Exclude invalid records from the target and track the dropped count.&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;LI&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;Fail:&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; Stop the affected flow when a record violates the rule.&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;/UL&gt;&lt;P&gt;&lt;SPAN&gt;For this fictional scenario, I would preserve the original order, explicitly route unresolved records to a quarantine dataset, and define how corrected records return to processing. A drop expectation does not automatically create that recovery path.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;I would also reconcile sales amounts across accepted and quarantined records before publishing totals. Passing row-level checks alone does not establish that a report is complete.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;How does your team decide when a failed quality rule should retain a sales record, quarantine it, or block processing?&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Reference: &lt;/SPAN&gt;&lt;A href="https://learn.microsoft.com/en-us/azure/databricks/ldp/expectations" target="_blank" rel="noopener"&gt;&lt;SPAN&gt;Azure Databricks pipeline expectations&lt;/SPAN&gt;&lt;/A&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;#Databricks #Lakeflow #DataQuality&lt;/SPAN&gt;&lt;/P&gt;</description>
    <pubDate>Wed, 04 Nov 2026 18:27:50 GMT</pubDate>
    <dc:creator>amitsharma1707</dc:creator>
    <dc:date>2026-11-04T18:27:50Z</dc:date>
    <item>
      <title>Lakeflow Expectations: Dropping Bad Rows Can Hide Real Sales</title>
      <link>https://community.databricks.com/t5/data-engineering/lakeflow-expectations-dropping-bad-rows-can-hide-real-sales/m-p/175627#M57025</link>
      <description>&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;A practical data quality lesson: choose the response to a failed rule based on its business impact.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Consider a hypothetical retailer using Azure Databricks. An incoming order has a valid order ID and sales amount, but its customer ID is missing.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;A rule such as &lt;/SPAN&gt;&lt;SPAN&gt;customer_id IS NOT NULL&lt;/SPAN&gt;&lt;SPAN&gt; catches the problem. However, dropping that row from the reporting dataset also removes its sales amount. The remaining records pass validation, while reported revenue becomes incomplete.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Lakeflow pipeline expectations offer three responses:&lt;/SPAN&gt;&lt;/P&gt;&lt;UL&gt;&lt;LI&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;Warn:&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; Keep invalid records and collect quality metrics.&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;LI&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;Drop:&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; Exclude invalid records from the target and track the dropped count.&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;LI&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;Fail:&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;SPAN&gt; Stop the affected flow when a record violates the rule.&lt;/SPAN&gt;&lt;/P&gt;&lt;/LI&gt;&lt;/UL&gt;&lt;P&gt;&lt;SPAN&gt;For this fictional scenario, I would preserve the original order, explicitly route unresolved records to a quarantine dataset, and define how corrected records return to processing. A drop expectation does not automatically create that recovery path.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;I would also reconcile sales amounts across accepted and quarantined records before publishing totals. Passing row-level checks alone does not establish that a report is complete.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&lt;SPAN&gt;How does your team decide when a failed quality rule should retain a sales record, quarantine it, or block processing?&lt;/SPAN&gt;&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Reference: &lt;/SPAN&gt;&lt;A href="https://learn.microsoft.com/en-us/azure/databricks/ldp/expectations" target="_blank" rel="noopener"&gt;&lt;SPAN&gt;Azure Databricks pipeline expectations&lt;/SPAN&gt;&lt;/A&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;#Databricks #Lakeflow #DataQuality&lt;/SPAN&gt;&lt;/P&gt;</description>
      <pubDate>Wed, 04 Nov 2026 18:27:50 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/lakeflow-expectations-dropping-bad-rows-can-hide-real-sales/m-p/175627#M57025</guid>
      <dc:creator>amitsharma1707</dc:creator>
      <dc:date>2026-11-04T18:27:50Z</dc:date>
    </item>
  </channel>
</rss>

