DeltaFileStatistics on a nested column (`created date.shipment`) cause filtering issues

Navinkumar_K
New Contributor

a month ago

Environment
- Databricks Runtime version: 17.3 LTS
- Cloud: Azure
- Catalog: Unity Catalog
- Table format: Delta

Summary
We have a Delta table named `shipment` with a column `created date.shipment` (a column whose name contains a dot). Delta collects deltafilestatistics on this column. When filtering on this column, the filter does not behave correctly — it appears that files are being incorrectly skipped, so queries return incomplete/wrong results.

Steps we took to investigate
1. In production, a query filtering on `created date.shipment` returned incorrect/incomplete results while deltafilestatistics is present on that column.
2. To isolate the issue, we cloned the table into our sandbox environment.
3. In the sandbox, we removed / disabled the file statistics on this column. After doing so, filtering on `created date.shipment` worked correctly.
4. However, on the next scheduled run, the table was overwritten (full refresh / overwrite write).
5. After that overwrite, filtering worked correctly even with the data-skipping statistics present again.

Working hypothesis
It looks like stale or incorrectly computed data-skipping statistics were causing files to be wrongly skipped during predicate pushdown. The full overwrite recomputed fresh statistics, which resolved the problem.

Example (illustrative)
- Table `shipment`, filter: WHERE created_date.shipment = '2026-01-15'
- Expected: N matching rows
- Actual (with stale stats): zero rows, because files containing matches were skipped
- After removing stats OR after overwrite: correct N rows returned

Questions
1. Is this a known issue with deltafilestatistics on column names containing a dot?
2. Can stale/incorrect statistics persist and cause wrong results until a full overwrite recomputes them? If so, is that expected?
3. What is the recommended way to force a recompute of data-skipping statistics WITHOUT a full table overwrite?

Reproducibility
We can reproduce the original incorrect behavior in our sandbox clone (before overwrite) and can provide query plans / EXPLAIN output and table history (DESCRIBE HISTORY) on request.

0 Kudos

iyashk-DB
Databricks Employee
Databricks Employee

Sunday

This is a known class of issue, not user error. Delta Lake stores min/max/null-count statistics in JSON using dot notation to represent nested struct paths (e.g. address.city means the city field inside an address struct). When a flat column's name contains a dot, like created date.shipment, the stats system can misinterpret it as a struct path reference. This can produce wrong statistics for that column, and since incorrect min/max values will cause the engine to skip files it shouldn't, you get incomplete query results — not just a missed optimization, but an actual correctness problem.

The reason your overwrite fixed it: rewriting the table recollects fresh statistics for all files, so the bogus stats are gone.

To fix it without overwriting, you have two options:

Option 1: Recompute statistics on all existing files (DBR 14.3+)

 
ANALYZE TABLE your_table_name COMPUTE DELTA STATISTICS
This backfills stats for every file currently in the table.

Option 2: Exclude that column from stats collection entirely

If the column isn't something you filter on for skipping purposes, just pull it out of stats collection:

 
ALTER TABLE your_table_name SET TBLPROPERTIES(
  'delta.dataSkippingStatsColumns' = 'col1,col2,col3' -- list every column you DO want stats on, omit the problematic one
);

ANALYZE TABLE your_table_name COMPUTE DELTA STATISTICS

You need the ANALYZE TABLE after the ALTER to rebuild stats on existing files. Changing the property alone only affects future writes.

Going forward, the safer long-term fix is to rename the column to something without a dot (and without the space). Column names with dots will always be fragile in contexts where Delta uses dot notation internally.