topic Re: Databricks Variant Data Type in Data Engineering

Databricks Variant Data Type

RobsonNLPT — Fri, 26 Jul 2024 10:21:14 GMT

I've tried to enabled a table to test the new variant data type (public preview)

I used the alter cmd:

ALTER TABLE tablexxxx SET TBLPROPERTIES('delta.feature.variantType-preview' = 'supported')

and I have the error

[DELTA_UNSUPPORTED_FEATURES_IN_CONFIG] Table feature(s) configured in the following Spark configs or Delta table properties are not recognized by this version of Databricks: delta.feature.varianttype-preview.

Any help?

Re: Databricks Variant Data Type

szymon_dybczak — Fri, 26 Jul 2024 11:15:53 GMT

Hi @RobsonNLPT ,

What runtime do you use?

Databricks Runtime 15.3 or above to read and write tables with variant support enabled.

Re: Databricks Variant Data Type

RobsonNLPT — Fri, 26 Jul 2024 11:23:40 GMT

Yes but I tried using 2 cluster types

SQL Serverless and New Compute Serverless.

Error in both

Re: Databricks Variant Data Type

szymon_dybczak — Fri, 26 Jul 2024 11:39:57 GMT

According to documentation, serverless in April was on runtime that roughly corresponds to 14.3. Since then, newest version is 2024.30, but they don't mention anywhere that runtime is equivalent to 15.3

Serverless compute release notes - Azure Databricks | Microsoft Learn

Re: Databricks Variant Data Type

RobsonNLPT — Fri, 26 Jul 2024 12:15:16 GMT

But wat about SQL Serverless?

Re: Databricks Variant Data Type

RobsonNLPT — Fri, 26 Jul 2024 12:22:33 GMT

I've tested using a notebook with a cluster and runtime 15.3.

I was able to create a table with variant data type using spark sql api

Re: Databricks Variant Data Type

szymon_dybczak — Fri, 26 Jul 2024 12:31:46 GMT

Yep, it succeeded because now you've used runtime that is supported. If the answer was helpful to you, you can check it as a solution.