<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic DeltaFileStatistics on a nested column (`created date.shipment`) cause filtering issues in Data Engineering</title>
    <link>https://community.databricks.com/t5/data-engineering/deltafilestatistics-on-a-nested-column-created-date-shipment/m-p/160169#M54863</link>
    <description>&lt;P&gt;Environment&lt;BR /&gt;- Databricks Runtime version: 17.3 LTS&lt;BR /&gt;- Cloud: Azure&lt;BR /&gt;- Catalog: Unity Catalog&lt;BR /&gt;- Table format: Delta&lt;/P&gt;&lt;P&gt;Summary&lt;BR /&gt;We have a Delta table named `shipment` with a column `created date.shipment` (a column whose name contains a dot). Delta collects deltafilestatistics on this column. When filtering on this column, the filter does not behave correctly — it appears that files are being incorrectly skipped, so queries return incomplete/wrong results.&lt;/P&gt;&lt;P&gt;Steps we took to investigate&lt;BR /&gt;1. In production, a query filtering on `created date.shipment` returned incorrect/incomplete results while deltafilestatistics is present on that column.&lt;BR /&gt;2. To isolate the issue, we cloned the table into our sandbox environment.&lt;BR /&gt;3. In the sandbox, we removed / disabled the file statistics on this column. After doing so, filtering on `created date.shipment` worked correctly.&lt;BR /&gt;4. However, on the next scheduled run, the table was overwritten (full refresh / overwrite write).&lt;BR /&gt;5. After that overwrite, filtering worked correctly even with the data-skipping statistics present again.&lt;/P&gt;&lt;P&gt;Working hypothesis&lt;BR /&gt;It looks like stale or incorrectly computed data-skipping statistics were causing files to be wrongly skipped during predicate pushdown. The full overwrite recomputed fresh statistics, which resolved the problem.&lt;/P&gt;&lt;P&gt;Example (illustrative)&lt;BR /&gt;- Table `shipment`, filter: WHERE created_date.shipment = '2026-01-15'&lt;BR /&gt;- Expected: N matching rows&lt;BR /&gt;- Actual (with stale stats): zero rows, because files containing matches were skipped&lt;BR /&gt;- After removing stats OR after overwrite: correct N rows returned&lt;/P&gt;&lt;P&gt;Questions&lt;BR /&gt;1. Is this a known issue with deltafilestatistics on column names containing a dot?&lt;BR /&gt;2. Can stale/incorrect statistics persist and cause wrong results until a full overwrite recomputes them? If so, is that expected?&lt;BR /&gt;3. What is the recommended way to force a recompute of data-skipping statistics WITHOUT a full table overwrite?&lt;BR /&gt;&lt;BR /&gt;&lt;/P&gt;&lt;P&gt;Reproducibility&lt;BR /&gt;We can reproduce the original incorrect behavior in our sandbox clone (before overwrite) and can provide query plans / EXPLAIN output and table history (DESCRIBE HISTORY) on request.&lt;/P&gt;</description>
    <pubDate>Tue, 23 Jun 2026 03:24:27 GMT</pubDate>
    <dc:creator>Navinkumar_K</dc:creator>
    <dc:date>2026-06-23T03:24:27Z</dc:date>
    <item>
      <title>DeltaFileStatistics on a nested column (`created date.shipment`) cause filtering issues</title>
      <link>https://community.databricks.com/t5/data-engineering/deltafilestatistics-on-a-nested-column-created-date-shipment/m-p/160169#M54863</link>
      <description>&lt;P&gt;Environment&lt;BR /&gt;- Databricks Runtime version: 17.3 LTS&lt;BR /&gt;- Cloud: Azure&lt;BR /&gt;- Catalog: Unity Catalog&lt;BR /&gt;- Table format: Delta&lt;/P&gt;&lt;P&gt;Summary&lt;BR /&gt;We have a Delta table named `shipment` with a column `created date.shipment` (a column whose name contains a dot). Delta collects deltafilestatistics on this column. When filtering on this column, the filter does not behave correctly — it appears that files are being incorrectly skipped, so queries return incomplete/wrong results.&lt;/P&gt;&lt;P&gt;Steps we took to investigate&lt;BR /&gt;1. In production, a query filtering on `created date.shipment` returned incorrect/incomplete results while deltafilestatistics is present on that column.&lt;BR /&gt;2. To isolate the issue, we cloned the table into our sandbox environment.&lt;BR /&gt;3. In the sandbox, we removed / disabled the file statistics on this column. After doing so, filtering on `created date.shipment` worked correctly.&lt;BR /&gt;4. However, on the next scheduled run, the table was overwritten (full refresh / overwrite write).&lt;BR /&gt;5. After that overwrite, filtering worked correctly even with the data-skipping statistics present again.&lt;/P&gt;&lt;P&gt;Working hypothesis&lt;BR /&gt;It looks like stale or incorrectly computed data-skipping statistics were causing files to be wrongly skipped during predicate pushdown. The full overwrite recomputed fresh statistics, which resolved the problem.&lt;/P&gt;&lt;P&gt;Example (illustrative)&lt;BR /&gt;- Table `shipment`, filter: WHERE created_date.shipment = '2026-01-15'&lt;BR /&gt;- Expected: N matching rows&lt;BR /&gt;- Actual (with stale stats): zero rows, because files containing matches were skipped&lt;BR /&gt;- After removing stats OR after overwrite: correct N rows returned&lt;/P&gt;&lt;P&gt;Questions&lt;BR /&gt;1. Is this a known issue with deltafilestatistics on column names containing a dot?&lt;BR /&gt;2. Can stale/incorrect statistics persist and cause wrong results until a full overwrite recomputes them? If so, is that expected?&lt;BR /&gt;3. What is the recommended way to force a recompute of data-skipping statistics WITHOUT a full table overwrite?&lt;BR /&gt;&lt;BR /&gt;&lt;/P&gt;&lt;P&gt;Reproducibility&lt;BR /&gt;We can reproduce the original incorrect behavior in our sandbox clone (before overwrite) and can provide query plans / EXPLAIN output and table history (DESCRIBE HISTORY) on request.&lt;/P&gt;</description>
      <pubDate>Tue, 23 Jun 2026 03:24:27 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/deltafilestatistics-on-a-nested-column-created-date-shipment/m-p/160169#M54863</guid>
      <dc:creator>Navinkumar_K</dc:creator>
      <dc:date>2026-06-23T03:24:27Z</dc:date>
    </item>
    <item>
      <title>Re: DeltaFileStatistics on a nested column (`created date.shipment`) cause filtering issues</title>
      <link>https://community.databricks.com/t5/data-engineering/deltafilestatistics-on-a-nested-column-created-date-shipment/m-p/163360#M55131</link>
      <description>&lt;DIV class="qMYqUG_convSearchResultHighlightRoot"&gt;
&lt;DIV class="" data-is-intersecting="true" data-turn-id-container="request-WEB:71d4366b-094a-44cc-b4da-5a592e56f096-3"&gt;
&lt;SECTION class="text-token-text-primary w-full focus:outline-none has-data-writing-block:pointer-events-none [&amp;amp;:has([data-writing-block])&amp;gt;*]:pointer-events-auto R6Vx5W_threadScrollVars scroll-mb-[calc(var(--scroll-root-safe-area-inset-bottom,0px)+var(--thread-response-height))] scroll-mt-[calc(var(--header-height)+min(200px,max(70px,20svh)))]" dir="auto" data-turn="assistant" data-testid="conversation-turn-6" data-turn-id-container="request-WEB:71d4366b-094a-44cc-b4da-5a592e56f096-3" data-turn-id="request-WEB:71d4366b-094a-44cc-b4da-5a592e56f096-3"&gt;
&lt;DIV class="text-base my-auto mx-auto pb-15 [--thread-content-margin:var(--thread-content-margin-xs,calc(var(--spacing)*4))] @w-sm/main:[--thread-content-margin:var(--thread-content-margin-sm,calc(var(--spacing)*6))] @w-lg/main:[--thread-content-margin:var(--thread-content-margin-lg,calc(var(--spacing)*16))] px-(--thread-content-margin)"&gt;
&lt;DIV class="[--thread-content-max-width:40rem] @w-lg/main:[--thread-content-max-width:48rem] mx-auto max-w-(--thread-content-max-width) flex-1 group/turn-messages focus-visible:outline-hidden relative flex w-full min-w-0 flex-col agent-turn" data-conversation-screenshot-content=""&gt;
&lt;DIV class="flex max-w-full flex-col gap-4 grow"&gt;
&lt;DIV class="min-h-8 text-message relative flex w-full flex-col items-end gap-2 text-start break-words whitespace-normal outline-none keyboard-focused:focus-ring [.text-message+&amp;amp;]:mt-1" dir="auto" tabindex="0" data-turn-start-message="true" data-message-model-slug="gpt-5-5" data-message-id="ff7d8aa8-471c-48ac-b307-1a201b22b919" data-message-author-role="assistant"&gt;
&lt;DIV class="flex w-full flex-col gap-1 empty:hidden"&gt;
&lt;DIV class="markdown prose dark:prose-invert wrap-break-word w-full dark markdown-new-styling"&gt;
&lt;P class="PDq2pG_selectionAnchorContainer" data-end="605" data-start="0"&gt;This is a known class of issue, not user error. Delta Lake stores min/max/null-count statistics in JSON using dot notation to represent nested struct paths (e.g. &lt;CODE data-end="176" data-start="162"&gt;address.city&lt;/CODE&gt; means the &lt;CODE data-end="193" data-start="187"&gt;city&lt;/CODE&gt; field inside an &lt;CODE data-end="219" data-start="210"&gt;address&lt;/CODE&gt; struct). When a &lt;EM data-end="242" data-start="236"&gt;flat&lt;/EM&gt; column's name contains a dot, like &lt;CODE data-end="301" data-start="278"&gt;created date.shipment&lt;/CODE&gt;, the stats system can misinterpret it as a struct path reference. This can produce wrong statistics for that column, and since incorrect min/max values will cause the engine to skip files it shouldn't, you get incomplete query results — not just a missed optimization, but an actual correctness problem.&lt;/P&gt;
&lt;P data-end="734" data-start="607"&gt;The reason your overwrite fixed it: rewriting the table recollects fresh statistics for all files, so the bogus stats are gone.&lt;/P&gt;
&lt;P data-end="788" data-start="736"&gt;To fix it without overwriting, you have two options:&lt;/P&gt;
&lt;P data-end="858" data-start="790"&gt;&lt;STRONG data-end="858" data-start="790"&gt;Option 1: Recompute statistics on all existing files (DBR 14.3+)&lt;/STRONG&gt;&lt;/P&gt;
&lt;DIV class="relative w-full mt-4 mb-1"&gt;
&lt;DIV class=""&gt;
&lt;DIV class="contents"&gt;
&lt;DIV class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"&gt;
&lt;DIV class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"&gt;
&lt;DIV class="pointer-events-none absolute inset-x-4 top-12 bottom-4"&gt;
&lt;DIV class="pointer-events-none sticky z-40 shrink-0 z-1!"&gt;
&lt;DIV class="sticky bg-token-border-light"&gt;&amp;nbsp;&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;DIV class="relative"&gt;
&lt;DIV class="h-full min-h-0 min-w-0"&gt;
&lt;DIV class="h-full min-h-0 min-w-0"&gt;
&lt;DIV class=""&gt;
&lt;DIV class="relative"&gt;
&lt;DIV class=""&gt;
&lt;DIV class="relative z-0 flex max-w-full"&gt;
&lt;DIV id="code-block-viewer" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼs ͼ16" dir="ltr"&gt;
&lt;DIV class="cm-scroller"&gt;
&lt;PRE class="cm-content q9tKkq_readonly m-0"&gt;&lt;CODE&gt;&lt;SPAN&gt;ANALYZE &lt;/SPAN&gt;&lt;SPAN class="ͼv"&gt;TABLE&lt;/SPAN&gt;&lt;SPAN&gt; your_table_name COMPUTE DELTA STATISTICS&lt;/SPAN&gt;&lt;/CODE&gt;&lt;/PRE&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;DIV class=""&gt;
&lt;DIV class=""&gt;&lt;SPAN&gt;This backfills stats for every file currently in the table.&lt;/SPAN&gt;&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;P data-end="1052" data-start="988"&gt;&lt;STRONG data-end="1052" data-start="988"&gt;Option 2: Exclude that column from stats collection entirely&lt;/STRONG&gt;&lt;/P&gt;
&lt;P data-end="1158" data-start="1054"&gt;If the column isn't something you filter on for skipping purposes, just pull it out of stats collection:&lt;/P&gt;
&lt;DIV class="relative w-full mt-4 mb-1"&gt;
&lt;DIV class=""&gt;
&lt;DIV class="contents"&gt;
&lt;DIV class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"&gt;
&lt;DIV class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"&gt;
&lt;DIV class="pointer-events-none absolute inset-x-4 top-12 bottom-4"&gt;
&lt;DIV class="pointer-events-none sticky z-40 shrink-0 z-1!"&gt;
&lt;DIV class="sticky bg-token-border-light"&gt;&amp;nbsp;&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;DIV class="relative"&gt;
&lt;DIV class="h-full min-h-0 min-w-0"&gt;
&lt;DIV class="h-full min-h-0 min-w-0"&gt;
&lt;DIV class=""&gt;
&lt;DIV class="relative"&gt;
&lt;DIV class=""&gt;
&lt;DIV class="relative z-0 flex max-w-full"&gt;
&lt;DIV id="code-block-viewer" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼs ͼ16" dir="ltr"&gt;
&lt;DIV class="cm-scroller"&gt;
&lt;PRE class="cm-content q9tKkq_readonly m-0"&gt;&lt;CODE&gt;&lt;SPAN class="ͼv"&gt;ALTER&lt;/SPAN&gt; &lt;SPAN class="ͼv"&gt;TABLE&lt;/SPAN&gt;&lt;SPAN&gt; your_table_name &lt;/SPAN&gt;&lt;SPAN class="ͼv"&gt;SET&lt;/SPAN&gt;&lt;SPAN&gt; TBLPROPERTIES(
  &lt;/SPAN&gt;&lt;SPAN class="ͼz"&gt;'delta.dataSkippingStatsColumns'&lt;/SPAN&gt; &lt;SPAN class="ͼv"&gt;=&lt;/SPAN&gt; &lt;SPAN class="ͼz"&gt;'col1,col2,col3'&lt;/SPAN&gt; &lt;SPAN class="ͼt"&gt;-- list every column you DO want stats on, omit the problematic one&lt;/SPAN&gt;&lt;SPAN&gt;
);

ANALYZE &lt;/SPAN&gt;&lt;SPAN class="ͼv"&gt;TABLE&lt;/SPAN&gt;&lt;SPAN&gt; your_table_name COMPUTE DELTA STATISTICS&lt;/SPAN&gt;&lt;/CODE&gt;&lt;/PRE&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;P data-end="1538" data-start="1400"&gt;You need the &lt;CODE data-end="1428" data-start="1413"&gt;ANALYZE TABLE&lt;/CODE&gt; after the &lt;CODE data-end="1446" data-start="1439"&gt;ALTER&lt;/CODE&gt; to rebuild stats on existing files. Changing the property alone only affects future writes.&lt;/P&gt;
&lt;P data-is-only-node="" data-is-last-node="" data-end="1754" data-start="1540"&gt;Going forward, the safer long-term fix is to rename the column to something without a dot (and without the space). Column names with dots will always be fragile in contexts where Delta uses dot notation internally.&lt;/P&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;
&lt;/SECTION&gt;
&lt;/DIV&gt;
&lt;/DIV&gt;</description>
      <pubDate>Sun, 19 Jul 2026 09:07:56 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/deltafilestatistics-on-a-nested-column-created-date-shipment/m-p/163360#M55131</guid>
      <dc:creator>iyashk-DB</dc:creator>
      <dc:date>2026-07-19T09:07:56Z</dc:date>
    </item>
  </channel>
</rss>

