DLT and Unity catalog best practices

[RicksDB]

What are the best practices when using DLT with a unity catalog aware workspace since it is not supported at the moment?

Do you create a copy of every table you need in the hive metastore in order to reference them? Or is it recommended to wait until support to migrate those tables?

Thanks!

[Pat]

That's a good one. I am trying to understand this part as I've been attending couple of Databricks sessions lately. Listening about the UC and DLT, but they keep this separate, first a lot about UC, UC, UC then you get info about DLT and Data Science Workspace integration, Feature store, but then you go to UC Limitations page to learn that is not currently supported with DLT.

Don't take me wrong both functionalities are great, but how should we use the both?

[Kaniz]

Hi @E H​ , We haven’t heard from you on the last response from @Pat Sienkiewicz​ . I hope you are doing well.

[osoucy]

From what I heard from the rep I'm working with, integrating UC with DLT is part of the roadmap, but not yet possible. They mentioned a few workarounds, but they are cumbersome and not really practical.

For now I stay away from UC and only use DLT until the two are compatible.

[pmarct]

Do you have any resources on those work arounds?

[Andrij_D]

Hi @E H​ , the best way to manage this for the time being is to limit DLT to logic that is considered 'internal' i.e., leveraging the default hive metastore. For now at least, this should be treated as a transient landing zone. From here you can refine, munge and transform the data meanwhile streaming that to UC catalogs for UC Sharing.

DLT should be treated as a process or task in a larger pipeline since it essentially couples materializations of your data together into logical segments. As a process this should be decoupled from your larger workflow anyway and should be allowed to operate asynchronously. Even when we integrate UC with DLT this principle should still be considered.