cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Understanding the effects of upgrading to Unity Catalog.

190809
Contributor

Hello, I have been trying to solidify my understanding of what changes occur after I upgrade my data to Unity Catalog. Our current ingestion process incrementally loads new data to tables in the hive metastore which are mounted to an s3 bucket. If I upgrade these tables to UC, will the daily incremental data also update in UC or is the data in UC just a 'moment in time' copy of my tables in the hive metatsore. Thanks in advance.

2 REPLIES 2

Anonymous
Not applicable

@Rachel Cunninghamโ€‹ :

If you upgrade your existing tables to Unity Catalog, then any new data that you incrementally load into those tables will also be reflected in Unity Catalog. Unity Catalog provides a metadata layer on top of your data, which means it maintains a logical view of your data and tracks changes to it, rather than being a separate copy of your data.

So, when you upgrade your tables to Unity Catalog, you are essentially migrating the metadata of those tables to Unity Catalog. This means that any changes to those tables, including incremental data loads, will be tracked by Unity Catalog. Unity Catalog maintains a transaction log, which records all changes made to the metadata in the catalog.

Therefore, if you load new data into your tables after upgrading them to Unity Catalog, those changes will be reflected in the metadata of Unity Catalog. However, it's worth noting that any changes to the underlying table structure (e.g. adding or dropping columns) will need to be made in the Hive metastore, and then synced to Unity Catalog using the SYNC METADATA command.

Anonymous
Not applicable

Hi @Rachel Cunninghamโ€‹ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group