Environment
- Databricks Runtime version: 17.3 LTS
- Cloud: Azure
- Catalog: Unity Catalog
- Table format: Delta
Summary
We have a Delta table named `shipment` with a column `created date.shipment` (a column whose name contains a dot). Delta collects deltafilestatistics on this column. When filtering on this column, the filter does not behave correctly — it appears that files are being incorrectly skipped, so queries return incomplete/wrong results.
Steps we took to investigate
1. In production, a query filtering on `created date.shipment` returned incorrect/incomplete results while deltafilestatistics is present on that column.
2. To isolate the issue, we cloned the table into our sandbox environment.
3. In the sandbox, we removed / disabled the file statistics on this column. After doing so, filtering on `created date.shipment` worked correctly.
4. However, on the next scheduled run, the table was overwritten (full refresh / overwrite write).
5. After that overwrite, filtering worked correctly even with the data-skipping statistics present again.
Working hypothesis
It looks like stale or incorrectly computed data-skipping statistics were causing files to be wrongly skipped during predicate pushdown. The full overwrite recomputed fresh statistics, which resolved the problem.
Example (illustrative)
- Table `shipment`, filter: WHERE created_date.shipment = '2026-01-15'
- Expected: N matching rows
- Actual (with stale stats): zero rows, because files containing matches were skipped
- After removing stats OR after overwrite: correct N rows returned
Questions
1. Is this a known issue with deltafilestatistics on column names containing a dot?
2. Can stale/incorrect statistics persist and cause wrong results until a full overwrite recomputes them? If so, is that expected?
3. What is the recommended way to force a recompute of data-skipping statistics WITHOUT a full table overwrite?
Reproducibility
We can reproduce the original incorrect behavior in our sandbox clone (before overwrite) and can provide query plans / EXPLAIN output and table history (DESCRIBE HISTORY) on request.
