This is a known class of issue, not user error. Delta Lake stores min/max/null-count statistics in JSON using dot notation to represent nested struct paths (e.g. address.city means the city field inside an address struct). When a flat column's name contains a dot, like created date.shipment, the stats system can misinterpret it as a struct path reference. This can produce wrong statistics for that column, and since incorrect min/max values will cause the engine to skip files it shouldn't, you get incomplete query results — not just a missed optimization, but an actual correctness problem.

The reason your overwrite fixed it: rewriting the table recollects fresh statistics for all files, so the bogus stats are gone.

To fix it without overwriting, you have two options:

Option 1: Recompute statistics on all existing files (DBR 14.3+)

ANALYZE TABLE your_table_name COMPUTE DELTA STATISTICS

Option 2: Exclude that column from stats collection entirely

If the column isn't something you filter on for skipping purposes, just pull it out of stats collection:

ALTER TABLE your_table_name SET TBLPROPERTIES(
  'delta.dataSkippingStatsColumns' = 'col1,col2,col3' -- list every column you DO want stats on, omit the problematic one
);

ANALYZE TABLE your_table_name COMPUTE DELTA STATISTICS

You need the ANALYZE TABLE after the ALTER to rebuild stats on existing files. Changing the property alone only affects future writes.

Going forward, the safer long-term fix is to rename the column to something without a dot (and without the space). Column names with dots will always be fragile in contexts where Delta uses dot notation internally.