Variant type table within DLT

[udi_azulay]

Hi,

I have a table with Variant type (preview) and works well in 15.3, when i try to run a code that reference this Variant type in a DLT pipeline i get : 

com.databricks.sql.transaction.tahoe.DeltaUnsupportedTableFeatureException: [DELTA_UNSUPPORTED_FEATURES_FOR_READ] Unsupported Delta read feature: table "trace_api_poc.ctv_decoder" requires reader table feature(s) that are unsupported by this version of Databricks: variantType-preview. Please refer to https://docs.microsoft.com/azure/databricks/delta/feature-compatibility for more information on Delta Lake feature compatibility.

when i try to change the policy and define Databricks Runtime Version to be 15.3, i get :

INVALID_PARAMETER_VALUE: [DLT ERROR CODE: INVALID_CLUSTER_SETTING.CLIENT_ERROR] The cluster policy specified in the pipeline settings is not compatible with Delta Live Tables. Remove 'spark_version’ from your cluster policy.
Please advise!

[szymon_dybczak]

Hi @udi_azulay ,

That's because DLT currently sits on lower version. Look at release notes: 

https://docs.databricks.com/en/release-notes/delta-live-tables/2024/22/index.html

Databricks Runtime versions used by this release

Channel:

CURRENT (default): Databricks Runtime 14.1

PREVIEW: Databricks Runtime 14.3

[thomas-totter]

Preview channel version currently is at 15.2. So we should be only one minor version increment away from variant being available in DLT (at least i hope so...).

[MAJVeld]

Now time has progressed, the current DLT runtime version (both CURRENT and PREVIEW) are based on Databricks Runtime version 15.4 as of December 2024. ([https://docs.databricks.com/en/release-notes/delta-live-tables/2024/49/index.html](https://docs.data...))

The VARIANT data type should thus be supported as it requires at least runtime version 15.3 ([https://docs.databricks.com/en/ingestion/variant.html)](https://docs.databricks.com/en/ingestion/var...)). As such, I expect variant columns to be supported when using a DLT pipeline. Even more, since the documentation does have no reference to unsupported data types in relation to DLT pipelines (https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/limitations)

However, when I define a DLT pipeline that ingests a message from Azure EventHub using the kafka connector, I get an error about unsupported features being used on the table. The error message suggests that this relates to an internal table as the name starts with __materialization_mat.... As a result, the table properties cannot be modified as suggested in the error message. Doing a full refresh of the tables associated with the pipeline does not resolve the issue.

As such I believe that VARIANT columns are still not supported in combination with DLT pipelines even though the runtime versions and documentation suggest otherwise.