Lakehosue table structure design

KuldeepChitraka
New Contributor III

We are in process of implementing a lakehouse using Azure Databricks. We already have a datalake in place

  • Azure Storage Datalake – Contains containers which has data in its native format.

How we are planning

  • Build Bronze layer by create bronze tables by reading data from datalake and storing it in bronze tables
    • Tables will enforce schema
    • There will be no partitioning on bronze tables
    • Each table will have _SourceFile   & _ingestionDate column in addition other columns
  • Silver Layer
    • Tables will contains data from bronze tables after applying transformation
    • Tables will have _loadDateTime column

What else column we should have in bronze & silver tables.

What should be our partition scheme. Can we partition tables in silver by LoadDateTime