Lakehosue table structure design
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-27-2022 06:22 AM
We are in process of implementing a lakehouse using Azure Databricks. We already have a datalake in place
- Azure Storage Datalake – Contains containers which has data in its native format.
How we are planning
- Build Bronze layer by create bronze tables by reading data from datalake and storing it in bronze tables
- Tables will enforce schema
- There will be no partitioning on bronze tables
- Each table will have _SourceFile & _ingestionDate column in addition other columns
- Silver Layer
- Tables will contains data from bronze tables after applying transformation
- Tables will have _loadDateTime column
What else column we should have in bronze & silver tables.
What should be our partition scheme. Can we partition tables in silver by LoadDateTime
Labels:
- Labels:
-
Azure databricks
-
Table