topic Can't enable "variantType-preview" using DLTs in Data Engineering

Can't enable "variantType-preview" using DLTs

Malthe — Fri, 17 Oct 2025 16:07:06 GMT

Using create_streaming_table and passing table properties as follows, I get an error running the pipeline for the first time:

> Your table schema requires manually enablement of the following table feature(s): variantType-preview.

I'm using this code:

create_streaming_table( name=name, schema=schema, table_properties={ "delta.feature.variantType-preview": "supported", "delta.parquet.compression.codec": "ZSTD", "pipelines.reset.allowed": "false", } )

That's a bit surprising because the table shows that this property was indeed set under the Unity Catalog table entry details. I believe this has been working in the past.

The workaround suggested (to manually enable it) does not work because the table was never refreshed:

> Streaming table [REDACTED] needs to be refreshed to execute QUERY. If the table is created from DBSQL, please run REFRESH STREAMING TABLE. If the table is created by a pipeline in Delta Live Tables, please run a pipeline update. SQLSTATE: 55019

Re: Can't enable "variantType-preview" using DLTs

ManojkMohan — Fri, 17 Oct 2025 16:24:16 GMT

@Malthe

Root Cause:

The property delta.feature.variantType-preview = supported tells Delta Lake to use the new Variant type, which is still in preview or limited availability

Why It Worked Previously

Earlier Databricks versions implicitly activated such features at table creation.
Recent updates added stricter activation controls

Solution Thinking:

https://learn.microsoft.com/en-us/azure/databricks/ldp/developer/ldp-python-ref-streaming-table

1. Clean Table Creation in Pipeline Code
If possible, delete the old table so your next pipeline run creates a new streaming table with all properties set and no legacy metadata issues. This works best when you control the pipeline end-to-end.

2. For Existing Tables (Manual Activation)
If your table already exists (created previously without the variantType flag fully activated), enable the feature manually using SQL

Immediately after Run

3. Final Step—Trigger Pipeline Update
If the table is under a Delta Live Tables pipeline, any schema or logic change will also trigger Databricks to re-activate its internal metadata:

Add or remove a comment in the create_streaming_table call, or

Edit and rerun your pipeline job.

This step triggers Databricks to finalize table activation and enable the new feature without manual SQL.

https://docs.databricks.com/aws/en/delta/variant

Be sure to use Databricks Runtime 15.3 or higher for full variant column support. If using DLT (Delta Live Tables), let the pipeline do the activation by deleting and recreating the table within the managed workflow.

Re: Can't enable "variantType-preview" using DLTs

Malthe — Fri, 17 Oct 2025 16:48:24 GMT

Unfortunately this is on a brand new table.

I was able to track down the runtime that successfully created and refreshed the table:

Worked:
dlt:16.4.10-delta-pipelines-dlt-release-dp-20251002-rc0-commit-6bcc9de-image-98a6a38
Did not work:
dlt:16.4.10-delta-pipelines-dlt-release-dp-20251009-rc0-commit-8c6b818-image-a20f551

Notice the October 2nd vs 9th release.

This could theoretically be an explanation, that we're seeing a regression here.

The problem persists whether I'm using the CURRENT or PREVIEW channel.

Re: Can't enable "variantType-preview" using DLTs

Malthe — Fri, 17 Oct 2025 17:49:31 GMT

Here's a self-contained example in SQL that demonstrates the issue:

CREATE TEMPORARY VIEW dummy_kafka AS SELECT 'id' AS id, 'topic' AS topic, current_timestamp() AS timestamp, 1::INTEGER AS partition, 1::long AS offset, parse_json('{}') AS body; CREATE STREAMING TABLE raw_telemetry ( id STRING PRIMARY KEY RELY, topic STRING NOT NULL, timestamp TIMESTAMP NOT NULL, partition INTEGER NOT NULL, offset LONG NOT NULL, body VARIANT NOT NULL ) TBLPROPERTIES ( 'delta.feature.variantType-preview' = 'supported' ) AS SELECT * FROM dummy_kafka;

Re: Can't enable "variantType-preview" using DLTs

Malthe — Fri, 17 Oct 2025 18:17:18 GMT

There's a workaround available in most situations which is to first create the table without the VARIANT column, run the pipeline at least once, and then add the column in a subsequent refresh.