Hi All,
I have 10 tables present on On-prem MS SQL DB and want to load 10 table data incrementally into Bronze delta table as append only. From Bronze to Silver , using merge query I want to load latest record into Silver delta table . Whatever latest record present in Source same record should present in Silver delta table while in bronze delta table I want to maintain the history means if any specific source record came as an update 4 times then I need 5 entry in bronze delta table and only latest record entry in Silver delta table which match with latest record present in Source. Also want to load all 10 tables dynamically into Bronze,Silver table using databricks. Could you please let me know how to approach on it to archive dynamically pass schema definition table load using single notebook either by maintain schema, table information in configuration table or excel file or any other simple way.
