Does enabling variantType-preview disable file-level data skipping for columns after a VARIANT colum

Alex13
New Contributor II

4 weeks ago - last edited 4 weeks ago

I have a Delta table (Unity Catalog, liquid clustering) where file-level data skipping never prunes files — a narrow predicate on the clustered timestamp column still reads every file (Files pruned = 0), even though row-group skipping works. The table has several VARIANT columns, and the timestamp column sits after them in the schema. dataSkippingStatsColumns includes the timestamp. Runtime is DBR 15.4+ (seen on 17.3 and 18.x).

I narrowed it down to the variantType-preview table feature. A freshly created table (which has only delta.feature.variantType) prunes correctly. The moment I enable variantType-preview on that same table, file pruning drops to zero:

-- fresh table: has only delta.feature.variantType, pruning works
CREATE TABLE t CLUSTER BY (event_time)
  TBLPROPERTIES ('delta.dataSkippingStatsColumns'='id,event_time')
  AS SELECT * FROM source;
-- query on a narrow event_time window -> Files pruned > 0

ALTER TABLE t SET TBLPROPERTIES ('delta.feature.variantType-preview'='supported');
-- same query -> Files pruned = 0

Placing the timestamp before the VARIANT columns makes pruning work even with variantType-preview enabled, so it looks specific to columns positioned after a VARIANT column.

A few things I noticed:

  • The affected table has both delta.feature.variantType and delta.feature.variantType-preview; the fresh one has only variantType.
  • ALTER TABLE ... DROP FEATURE 'variantType-preview' fails because VARIANT columns exist.
  • ANALYZE ... COMPUTE DELTA STATISTICS and OPTIMIZE FULL do not fix it; only a full rewrite does.

Questions:

  1. Is it expected that variantType-preview disables file-level skipping for columns after a VARIANT column, even when they are listed in dataSkippingStatsColumns?
  2. What is the difference between variantType and variantType-preview, and why would a table have both?
  3. Since DROP FEATURE is blocked by the VARIANT columns, is there a supported way off variantType-preview without a full table rewrite?

Thanks.

Labels:
0 Kudos

Louis_Frolio
Databricks Employee
Databricks Employee

Thursday

Hi @Alex13 , thanks for the detailed minimal repro. Isolating variantType-preview as the trigger was the hard part, and it makes your three questions answerable. Here is my read.

  1. Is this expected? Not according to anything public. The only documented statistics limitation is that VARIANT columns themselves do not provide minValues or maxValues. Data skipping is based on per-file statistics, and delta.dataSkippingStatsColumns exists specifically to name the scalar columns that should get them. Nothing in the docs says scalar columns positioned after a VARIANT column lose file-level statistics. Since event_time is explicitly listed and moving it before the VARIANT columns restores pruning, this looks like a preview-feature interaction or regression, not a normal VARIANT limitation. Your own evidence points at the read side rather than stale statistics: if the stats were simply missing from the files, ANALYZE ... COMPUTE DELTA STATISTICS or OPTIMIZE FULL would have fixed it. You can confirm by inspecting the add actions in the _delta_log JSON for the affected files. If minValues and maxValues for event_time are present but the scan still reads every file, the reader is mishandling the stats schema when variantType-preview is on the table. Either way, please open a Databricks Support case with this repro, the exact DBR build, DESCRIBE DETAIL output, table properties, and the query profile. Engineering does not pick up bug reports from Community threads, and this one deserves eyes.

  2. The difference between the two features. variantType-preview is the legacy table-feature name from the Public Preview period (DBR 15.3 era). variantType without the suffix is the current stable feature from the open Delta protocol, and the Delta Kernel changelog tracks them as separate reader/writer features. They are not two independent performance modes. A table showing both has either carried forward the older preview metadata or, as in your repro, had the preview property set explicitly on a modern table. Both being present is legal at the protocol level. The practical takeaway from your own experiment: on a current runtime, never set delta.feature.variantType-preview manually. Creating a VARIANT column already enables everything you need.

  3. Getting off variantType-preview. I would not keep retrying DROP FEATURE as an in-place fix. Dropping a feature requires removing all traces of it from the data files and transaction log before the protocol can downgrade, the variant features are not in the documented list of removable features, and I could not find a supported command that removes the legacy feature while retaining your VARIANT data. DEEP CLONE will not help either, since clones inherit the source protocol. The practical supported path is the one you found: create a replacement table on DBR 17.3 or above (CTAS, backfill, swap names), which lands you on the current feature model only. There is a silver lining. New tables on 17.3+ get variant shredding automatically, and DBR 18.1+ is recommended for variant statistics collection and data skipping, so the rewrite buys you real performance beyond fixing this issue.

Until the root cause is confirmed, a few things to try on a test copy:

  • Keep event_time (and any other skipping or clustering columns) ahead of the VARIANT columns in the schema.
  • If a full rewrite is too expensive right now, test ALTER TABLE ... ALTER COLUMN event_time FIRST as a logical reorder. Note this requires column mapping (delta.columnMapping.mode = name), so check that first. Treat it as an experiment, not a documented fix.
  • Keep delta.dataSkippingStatsColumns explicitly set to include event_time, recompute statistics after any property or schema change, and confirm the result in the query profile or EXPLAIN.

The fact that behavior changes with the feature flag and with column order is strong evidence of a product issue worth escalating. If you do open a ticket, please circle back here with what support says. Others will hit this.

Public references:

Regards, Louis

0 Kudos