cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

Understanding the effects of upgrading to Unity Catalog.

190809
Contributor

Hello, I have been trying to solidify my understanding of what changes occur after I upgrade my data to Unity Catalog. Our current ingestion process incrementally loads new data to tables in the hive metastore which are mounted to an s3 bucket. If I upgrade these tables to UC, will the daily incremental data also update in UC or is the data in UC just a 'moment in time' copy of my tables in the hive metatsore. Thanks in advance.

2 REPLIES 2

Anonymous
Not applicable

@Rachel Cunningham​ :

If you upgrade your existing tables to Unity Catalog, then any new data that you incrementally load into those tables will also be reflected in Unity Catalog. Unity Catalog provides a metadata layer on top of your data, which means it maintains a logical view of your data and tracks changes to it, rather than being a separate copy of your data.

So, when you upgrade your tables to Unity Catalog, you are essentially migrating the metadata of those tables to Unity Catalog. This means that any changes to those tables, including incremental data loads, will be tracked by Unity Catalog. Unity Catalog maintains a transaction log, which records all changes made to the metadata in the catalog.

Therefore, if you load new data into your tables after upgrading them to Unity Catalog, those changes will be reflected in the metadata of Unity Catalog. However, it's worth noting that any changes to the underlying table structure (e.g. adding or dropping columns) will need to be made in the Hive metastore, and then synced to Unity Catalog using the SYNC METADATA command.

Anonymous
Not applicable

Hi @Rachel Cunningham​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!