โ08-29-2022 05:43 PM
What are the best practices when using DLT with a unity catalog aware workspace since it is not supported at the moment?
Do you create a copy of every table you need in the hive metastore in order to reference them? Or is it recommended to wait until support to migrate those tables?
Thanks!
โ08-30-2022 01:10 AM
That's a good one. I am trying to understand this part as I've been attending couple of Databricks sessions lately. Listening about the UC and DLT, but they keep this separate, first a lot about UC, UC, UC then you get info about DLT and Data Science Workspace integration, Feature store, but then you go to UC Limitations page to learn that is not currently supported with DLT.
Don't take me wrong both functionalities are great, but how should we use the both?
โ11-05-2022 06:35 PM
From what I heard from the rep I'm working with, integrating UC with DLT is part of the roadmap, but not yet possible. They mentioned a few workarounds, but they are cumbersome and not really practical.
For now I stay away from UC and only use DLT until the two are compatible.
โ12-01-2022 09:40 AM
Do you have any resources on those work arounds?
โ01-04-2023 09:09 AM
Hi @E Hโ , the best way to manage this for the time being is to limit DLT to logic that is considered 'internal' i.e., leveraging the default hive metastore. For now at least, this should be treated as a transient landing zone. From here you can refine, munge and transform the data meanwhile streaming that to UC catalogs for UC Sharing.
DLT should be treated as a process or task in a larger pipeline since it essentially couples materializations of your data together into logical segments. As a process this should be decoupled from your larger workflow anyway and should be allowed to operate asynchronously. Even when we integrate UC with DLT this principle should still be considered.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group