We are implementing a lakehouse architecture and using Notebook to transform data from object storage. Most of the time our source is database for which there one folder per table in object storage.

We have structure like below for various notebooks

• GOLD (Folder)
• SILVER (Folder)
• BRONZE(Folder)
  • MasterRawNotebooks.py (Notebook)
    • Bronze_tables (Folder)
      • Table1_notebook.py
      • Table2_notebook.py

RAW folder contains notebook for each table which reads data from object storage and create DELTA table.

MasterRawNotebook contains one cell for each raw notebook which calls Table1_notebook, Table2_Notenook using %run

So when we execute MasterRawNotebook it runs the each notebook one by one and created table in database in databricks.

• Is it a right approach?
• We are are creating notebook for each table
• Or should execute BRONZE notebooks parallelly.
• How have you implemented the notebook pipeline when you are implementing Lakehouse Architecture.
• What kind of exception handling have you done in notebooks while loading data from BRONZE to SILVER
• If possible would you share folder structure , how notebooks are organized for loading , transformation etc.
• Any best practices to refer?