https://community.databricks.com/t5/data-engineering/deltafilestatistics-on-a-nested-column-created-date-shipment/m-p/160169#M54863 <P>Environment<BR />- Databricks Runtime version: 17.3 LTS<BR />- Cloud: Azure<BR />- Catalog: Unity Catalog<BR />- Table format: Delta</P><P>Summary<BR />We have a Delta table named `shipment` with a column `created date.shipment` (a column whose name contains a dot). Delta collects deltafilestatistics on this column. When filtering on this column, the filter does not behave correctly — it appears that files are being incorrectly skipped, so queries return incomplete/wrong results.</P><P>Steps we took to investigate<BR />1. In production, a query filtering on `created date.shipment` returned incorrect/incomplete results while deltafilestatistics is present on that column.<BR />2. To isolate the issue, we cloned the table into our sandbox environment.<BR />3. In the sandbox, we removed / disabled the file statistics on this column. After doing so, filtering on `created date.shipment` worked correctly.<BR />4. However, on the next scheduled run, the table was overwritten (full refresh / overwrite write).<BR />5. After that overwrite, filtering worked correctly even with the data-skipping statistics present again.</P><P>Working hypothesis<BR />It looks like stale or incorrectly computed data-skipping statistics were causing files to be wrongly skipped during predicate pushdown. The full overwrite recomputed fresh statistics, which resolved the problem.</P><P>Example (illustrative)<BR />- Table `shipment`, filter: WHERE created_date.shipment = '2026-01-15'<BR />- Expected: N matching rows<BR />- Actual (with stale stats): zero rows, because files containing matches were skipped<BR />- After removing stats OR after overwrite: correct N rows returned</P><P>Questions<BR />1. Is this a known issue with deltafilestatistics on column names containing a dot?<BR />2. Can stale/incorrect statistics persist and cause wrong results until a full overwrite recomputes them? If so, is that expected?<BR />3. What is the recommended way to force a recompute of data-skipping statistics WITHOUT a full table overwrite?<BR /><BR /></P><P>Reproducibility<BR />We can reproduce the original incorrect behavior in our sandbox clone (before overwrite) and can provide query plans / EXPLAIN output and table history (DESCRIBE HISTORY) on request.</P> Tue, 23 Jun 2026 03:24:27 GMT Navinkumar_K 2026-06-23T03:24:27Z https://community.databricks.com/t5/data-engineering/deltafilestatistics-on-a-nested-column-created-date-shipment/m-p/160169#M54863 <P>Environment<BR />- Databricks Runtime version: 17.3 LTS<BR />- Cloud: Azure<BR />- Catalog: Unity Catalog<BR />- Table format: Delta</P><P>Summary<BR />We have a Delta table named `shipment` with a column `created date.shipment` (a column whose name contains a dot). Delta collects deltafilestatistics on this column. When filtering on this column, the filter does not behave correctly — it appears that files are being incorrectly skipped, so queries return incomplete/wrong results.</P><P>Steps we took to investigate<BR />1. In production, a query filtering on `created date.shipment` returned incorrect/incomplete results while deltafilestatistics is present on that column.<BR />2. To isolate the issue, we cloned the table into our sandbox environment.<BR />3. In the sandbox, we removed / disabled the file statistics on this column. After doing so, filtering on `created date.shipment` worked correctly.<BR />4. However, on the next scheduled run, the table was overwritten (full refresh / overwrite write).<BR />5. After that overwrite, filtering worked correctly even with the data-skipping statistics present again.</P><P>Working hypothesis<BR />It looks like stale or incorrectly computed data-skipping statistics were causing files to be wrongly skipped during predicate pushdown. The full overwrite recomputed fresh statistics, which resolved the problem.</P><P>Example (illustrative)<BR />- Table `shipment`, filter: WHERE created_date.shipment = '2026-01-15'<BR />- Expected: N matching rows<BR />- Actual (with stale stats): zero rows, because files containing matches were skipped<BR />- After removing stats OR after overwrite: correct N rows returned</P><P>Questions<BR />1. Is this a known issue with deltafilestatistics on column names containing a dot?<BR />2. Can stale/incorrect statistics persist and cause wrong results until a full overwrite recomputes them? If so, is that expected?<BR />3. What is the recommended way to force a recompute of data-skipping statistics WITHOUT a full table overwrite?<BR /><BR /></P><P>Reproducibility<BR />We can reproduce the original incorrect behavior in our sandbox clone (before overwrite) and can provide query plans / EXPLAIN output and table history (DESCRIBE HISTORY) on request.</P> Tue, 23 Jun 2026 03:24:27 GMT https://community.databricks.com/t5/data-engineering/deltafilestatistics-on-a-nested-column-created-date-shipment/m-p/160169#M54863 Navinkumar_K 2026-06-23T03:24:27Z