Unity Catalog: Metastore 3 level Hierarchy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-02-2024 12:21 AM
I have data files categorized by application and region. Want to know the best way to load them into the Bronze and Silver layers while maintaining proper segregation.
For example, in our landing zone, we have a structure of raw files to be loaded using Autoloader with files from multiple applications and regions as in below. With a three-level naming convention in Unity catalog, I am concerned about difficulty in tracing table to file mapping in the future. Can you suggest the best possible architecture for this and explain its merits and internal organization? Thank you for your time and assistance.
Eg : Below is landing zone structure of raw files to be loaded using Autoloader.
App1 : Region1 : File1
App1 : Region1 : File2
App2 : Region1 : File1
App2 : Region1 : File2
There are multiple applications with around 11 regions and approx 100 files daily to be loaded