- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-23-2025 06:19 AM - edited 01-23-2025 06:38 AM
Now time has progressed, the current DLT runtime version (both CURRENT and PREVIEW) are based on Databricks Runtime version 15.4 as of December 2024. ([https://docs.databricks.com/en/release-notes/delta-live-tables/2024/49/index.html](https://docs.data...))
The VARIANT data type should thus be supported as it requires at least runtime version 15.3 ([https://docs.databricks.com/en/ingestion/variant.html)](https://docs.databricks.com/en/ingestion/var...)). As such, I expect variant columns to be supported when using a DLT pipeline. Even more, since the documentation does have no reference to unsupported data types in relation to DLT pipelines (https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/limitations)
However, when I define a DLT pipeline that ingests a message from Azure EventHub using the kafka connector, I get an error about unsupported features being used on the table. The error message suggests that this relates to an internal table as the name starts with __materialization_mat.... As a result, the table properties cannot be modified as suggested in the error message. Doing a full refresh of the tables associated with the pipeline does not resolve the issue.
As such I believe that VARIANT columns are still not supported in combination with DLT pipelines even though the runtime versions and documentation suggest otherwise.