topic Re: Variant type table within DLT in Data Engineering

Variant type table within DLT

udi_azulay — Thu, 01 Aug 2024 04:36:58 GMT

Hi,

I have a table with Variant type (preview) and works well in 15.3, when i try to run a code that reference this Variant type in a DLT pipeline i get :

com.databricks.sql.transaction.tahoe.DeltaUnsupportedTableFeatureException: [DELTA_UNSUPPORTED_FEATURES_FOR_READ] Unsupported Delta read feature: table "trace_api_poc.ctv_decoder" requires reader table feature(s) that are unsupported by this version of Databricks: variantType-preview. Please refer to https://docs.microsoft.com/azure/databricks/delta/feature-compatibility for more information on Delta Lake feature compatibility.

when i try to change the policy and define Databricks Runtime Version to be 15.3, i get :

INVALID_PARAMETER_VALUE: [DLT ERROR CODE: INVALID_CLUSTER_SETTING.CLIENT_ERROR] The cluster policy specified in the pipeline settings is not compatible with Delta Live Tables. Remove 'spark_version’ from your cluster policy.
Please advise!

Re: Variant type table within DLT

szymon_dybczak — Thu, 01 Aug 2024 07:44:26 GMT

Hi @udi_azulay ,

That's because DLT currently sits on lower version. Look at release notes:

https://docs.databricks.com/en/release-notes/delta-live-tables/2024/22/index.html

Databricks Runtime versions used by this release

Channel:

CURRENT (default): Databricks Runtime 14.1

PREVIEW: Databricks Runtime 14.3

Re: Variant type table within DLT

thomas-cap — Tue, 13 Aug 2024 11:34:29 GMT

Preview channel version currently is at 15.2. So we should be only one minor version increment away from variant being available in DLT (at least i hope so...).

Re: Variant type table within DLT

MAJVeld — Thu, 23 Jan 2025 14:38:29 GMT

Now time has progressed, the current DLT runtime version (both CURRENT and PREVIEW) are based on Databricks Runtime version 15.4 as of December 2024. ([https://docs.databricks.com/en/release-notes/delta-live-tables/2024/49/index.html](https://docs.databricks.com/en/release-notes/delta-live-tables/2024/49/index.html))

The VARIANT data type should thus be supported as it requires at least runtime version 15.3 ([https://docs.databricks.com/en/ingestion/variant.html)](https://docs.databricks.com/en/ingestion/variant.html)). As such, I expect variant columns to be supported when using a DLT pipeline. Even more, since the documentation does have no reference to unsupported data types in relation to DLT pipelines (https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/limitations)

However, when I define a DLT pipeline that ingests a message from Azure EventHub using the kafka connector, I get an error about unsupported features being used on the table. The error message suggests that this relates to an internal table as the name starts with __materialization_mat.... As a result, the table properties cannot be modified as suggested in the error message. Doing a full refresh of the tables associated with the pipeline does not resolve the issue.

As such I believe that VARIANT columns are still not supported in combination with DLT pipelines even though the runtime versions and documentation suggest otherwise.

Re: Variant type table within DLT

HNoa — Thu, 27 Feb 2025 15:47:33 GMT

I will echo this. I can run my materialized view when it's not attached to the pipeline (as long as I have this: "

TBLPROPERTIES ('delta.feature.variantType-preview' = 'supported')"), but as soon as we attach it to a pipeline it fails and says that we have a scala match error.

Re: Variant type table within DLT

HNoa — Thu, 27 Feb 2025 16:27:36 GMT

@MAJVeld We got it to work.
Let me know if this works for you. We're not using serverless right now, so I can't speak to that, but I had to set the channel to preview instead of current for it to work (and we're using photon, but that probably doesn't matter).

And again, after the create statement we had to set the tbl properties. Sounds like the table properties are what matter to you. You should be able to set it within the table definition. See screenshot.

Re: Variant type table within DLT

MAJVeld — Mon, 03 Mar 2025 12:37:02 GMT

I can indeed confirm that adding some additional table properties to the @Dlt attribute in the DLT pipeline definition resolved the earlier issues. Thanks for pointing this out.