I have a Delta table (Unity Catalog, liquid clustering) where file-level data skipping never prunes files — a narrow predicate on the clustered timestamp column still reads every file (Files pruned = 0), even though row-group skipping works. The table has several VARIANT columns, and the timestamp column sits after them in the schema. dataSkippingStatsColumns includes the timestamp. Runtime is DBR 15.4+ (seen on 17.3 and 18.x).
I narrowed it down to the variantType-preview table feature. A freshly created table (which has only delta.feature.variantType) prunes correctly. The moment I enable variantType-preview on that same table, file pruning drops to zero:
-- fresh table: has only delta.feature.variantType, pruning works
CREATE TABLE t CLUSTER BY (event_time)
TBLPROPERTIES ('delta.dataSkippingStatsColumns'='id,event_time')
AS SELECT * FROM source;
-- query on a narrow event_time window -> Files pruned > 0
ALTER TABLE t SET TBLPROPERTIES ('delta.feature.variantType-preview'='supported');
-- same query -> Files pruned = 0Placing the timestamp before the VARIANT columns makes pruning work even with variantType-preview enabled, so it looks specific to columns positioned after a VARIANT column.
A few things I noticed:
- The affected table has both delta.feature.variantType and delta.feature.variantType-preview; the fresh one has only variantType.
- ALTER TABLE ... DROP FEATURE 'variantType-preview' fails because VARIANT columns exist.
- ANALYZE ... COMPUTE DELTA STATISTICS and OPTIMIZE FULL do not fix it; only a full rewrite does.
Questions:
- Is it expected that variantType-preview disables file-level skipping for columns after a VARIANT column, even when they are listed in dataSkippingStatsColumns?
- What is the difference between variantType and variantType-preview, and why would a table have both?
- Since DROP FEATURE is blocked by the VARIANT columns, is there a supported way off variantType-preview without a full table rewrite?
Thanks.
