cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT and Unity catalog best practices

RicksDB
Contributor II

What are the best practices when using DLT with a unity catalog aware workspace since it is not supported at the moment?

Do you create a copy of every table you need in the hive metastore in order to reference them? Or is it recommended to wait until support to migrate those tables?

Thanks!

5 REPLIES 5

Pat
Honored Contributor III

That's a good one. I am trying to understand this part as I've been attending couple of Databricks sessions lately. Listening about the UC and DLT, but they keep this separate, first a lot about UC, UC, UC then you get info about DLT and Data Science Workspace integration, Feature store, but then you go to UC Limitations page to learn that is not currently supported with DLT.

Don't take me wrong both functionalities are great, but how should we use the both?

Kaniz_Fatma
Community Manager
Community Manager

Hi @E H​ , We haven’t heard from you on the last response from @Pat Sienkiewicz​ . I hope you are doing well.

osoucy
New Contributor II

From what I heard from the rep I'm working with, integrating UC with DLT is part of the roadmap, but not yet possible. They mentioned a few workarounds, but they are cumbersome and not really practical.

For now I stay away from UC and only use DLT until the two are compatible.

pmarct
New Contributor II

Do you have any resources on those work arounds?

Andrij_D
New Contributor II

Hi @E H​ , the best way to manage this for the time being is to limit DLT to logic that is considered 'internal' i.e., leveraging the default hive metastore. For now at least, this should be treated as a transient landing zone. From here you can refine, munge and transform the data meanwhile streaming that to UC catalogs for UC Sharing.

DLT should be treated as a process or task in a larger pipeline since it essentially couples materializations of your data together into logical segments. As a process this should be decoupled from your larger workflow anyway and should be allowed to operate asynchronously. Even when we integrate UC with DLT this principle should still be considered.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!