cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

What are the Delta Live Tables limitations with relation to Unity Catalog?

eimis_pacheco
Contributor

Hi community!

I was in a Databricks webinar and one of the participants said "Delta Live Tables seems to have some limitations when using with Unity Catalog. Is the idea to get parity with Hive?" and someone answered "DLT + Unity Catalog combination has two major limitations as of now : Not able to follow medallion architecture & no support for 3rd party library"

To what does he refer when he said "DLT not being able to follow medallion architecture & no support for 3rd party libraries?

Why does DLT struggle with Unity Catalog?

What are other limitations we should take into consideration before creating a workflow with DLT?

Thank you very much for your advice and insights

Regards

1 ACCEPTED SOLUTION

Accepted Solutions

Kaniz_Fatma
Community Manager
Community Manager

Hi @eimis_pachecoThe participant's statement refers to two specific limitations when using Delta Live Tables (DLT) with Unity Catalog:

1. DLT not being able to follow the medallion architecture: The Medallion architecture is a data management strategy that organizes data into tiers (bronze, silver, gold) based on the level of transformation and refinement. The limitation here implies that DLT, when used with Unity Catalog, may not support this tiered data management approach effectively.

2. No support for third party libraries: This means that DLT, in combination with Unity Catalog, does not currently support the integration or use of third-party libraries. This could limit the functionality and flexibility of your data workflows, as you may not be able to utilize specific tools or features provided by these libraries. As for why DLT struggles with the Unity Catalog, the specifics are not detailed in the provided sources. However, it could be due to technical constraints, compatibility issues, or architectural differences between DLT and Unity Catalog. 

Other limitations to consider when creating a workflow with DLT include:

- Only tables and views stored in a metastore can be shared using DLT.
- Only tables in Delta format are supported.
- View sharing is supported only in Databricks-to-Databricks sharing.
- Shareable views must be defined on Delta tables or other shareable views.
- There are limits on the number of files in metadata allowed for a shared table.
- Schemas named information_schema cannot be imported into a metastore because that schema name is reserved.

View solution in original post

2 REPLIES 2

Kaniz_Fatma
Community Manager
Community Manager

Hi @eimis_pachecoThe participant's statement refers to two specific limitations when using Delta Live Tables (DLT) with Unity Catalog:

1. DLT not being able to follow the medallion architecture: The Medallion architecture is a data management strategy that organizes data into tiers (bronze, silver, gold) based on the level of transformation and refinement. The limitation here implies that DLT, when used with Unity Catalog, may not support this tiered data management approach effectively.

2. No support for third party libraries: This means that DLT, in combination with Unity Catalog, does not currently support the integration or use of third-party libraries. This could limit the functionality and flexibility of your data workflows, as you may not be able to utilize specific tools or features provided by these libraries. As for why DLT struggles with the Unity Catalog, the specifics are not detailed in the provided sources. However, it could be due to technical constraints, compatibility issues, or architectural differences between DLT and Unity Catalog. 

Other limitations to consider when creating a workflow with DLT include:

- Only tables and views stored in a metastore can be shared using DLT.
- Only tables in Delta format are supported.
- View sharing is supported only in Databricks-to-Databricks sharing.
- Shareable views must be defined on Delta tables or other shareable views.
- There are limits on the number of files in metadata allowed for a shared table.
- Schemas named information_schema cannot be imported into a metastore because that schema name is reserved.

Hi @Kaniz_Fatma ,

Thank you for your answer. Wondering then why Databricks recommends the usage of both solutions (DLT and Unity Catalog) even with the existence of this imcompatibilities?

Best Regards

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!