Understanding the effects of upgrading to Unity Catalog.

[190809]

Hello, I have been trying to solidify my understanding of what changes occur after I upgrade my data to Unity Catalog. Our current ingestion process incrementally loads new data to tables in the hive metastore which are mounted to an s3 bucket. If I upgrade these tables to UC, will the daily incremental data also update in UC or is the data in UC just a 'moment in time' copy of my tables in the hive metatsore. Thanks in advance.

[Anonymous]

@Rachel Cunningham​ :

If you upgrade your existing tables to Unity Catalog, then any new data that you incrementally load into those tables will also be reflected in Unity Catalog. Unity Catalog provides a metadata layer on top of your data, which means it maintains a logical view of your data and tracks changes to it, rather than being a separate copy of your data.

So, when you upgrade your tables to Unity Catalog, you are essentially migrating the metadata of those tables to Unity Catalog. This means that any changes to those tables, including incremental data loads, will be tracked by Unity Catalog. Unity Catalog maintains a transaction log, which records all changes made to the metadata in the catalog.

Therefore, if you load new data into your tables after upgrading them to Unity Catalog, those changes will be reflected in the metadata of Unity Catalog. However, it's worth noting that any changes to the underlying table structure (e.g. adding or dropping columns) will need to be made in the Hive metastore, and then synced to Unity Catalog using the SYNC METADATA command.

[Anonymous]

Hi @Rachel Cunningham​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!