- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-22-2022 01:53 AM
With lakes and Hive metastore (external tables) I did it same way.
- landing storage with containers based on the source system/vendor separated from my lake to enable better authorization model to push data by system/vendor
- bronze container, like staging. dirs by system for delta table of all not transformed data from landing.
- silver. parsed/cleansed data in delta format, still in system and their objects hierarchy.
- silver. enhanced data? == my core data model, -> entities (product, customer...)
But the way I see it nowadays:
Do you already use Unity Catalog? Is this still question there? you are more and more forced there to use managed tables. you do not more care about structure of your lake / lakehouse. It is still more and more DDL representation of data like DWH. You create the structure in your Metastore *** UC managed location (it use ids to store your tables in a storage not human readable paths).
So now the question is more how to organize your Metastore ( catalogs, databases, tables) to follow this medallion arch. then how to structure your lake containers/directories ..in my opinion.