cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

DLT and Unity catalog best practices

RicksDB
Contributor II

What are the best practices when using DLT with a unity catalog aware workspace since it is not supported at the moment?

Do you create a copy of every table you need in the hive metastore in order to reference them? Or is it recommended to wait until support to migrate those tables?

Thanks!

4 REPLIES 4

Pat
Honored Contributor III

That's a good one. I am trying to understand this part as I've been attending couple of Databricks sessions lately. Listening about the UC and DLT, but they keep this separate, first a lot about UC, UC, UC then you get info about DLT and Data Science Workspace integration, Feature store, but then you go to UC Limitations page to learn that is not currently supported with DLT.

Don't take me wrong both functionalities are great, but how should we use the both?

osoucy
New Contributor II

From what I heard from the rep I'm working with, integrating UC with DLT is part of the roadmap, but not yet possible. They mentioned a few workarounds, but they are cumbersome and not really practical.

For now I stay away from UC and only use DLT until the two are compatible.

pmarct
New Contributor II

Do you have any resources on those work arounds?

Andrij_D
New Contributor II

Hi @E Hโ€‹ , the best way to manage this for the time being is to limit DLT to logic that is considered 'internal' i.e., leveraging the default hive metastore. For now at least, this should be treated as a transient landing zone. From here you can refine, munge and transform the data meanwhile streaming that to UC catalogs for UC Sharing.

DLT should be treated as a process or task in a larger pipeline since it essentially couples materializations of your data together into logical segments. As a process this should be decoupled from your larger workflow anyway and should be allowed to operate asynchronously. Even when we integrate UC with DLT this principle should still be considered.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group