delta table separate gold zone by different tenant

[alexgv12]

Hello, currently we have a process that builds with delta table the zones of bronze, silver and when it reaches gold we must create specific zones for each client because the schema changes, for this we create databases and separate tables, but when we are doing this process it takes a long time and in many occasions communication is lost in the notebook and the execution fails, what good practice, or alternatives could we do to improve our flow performance and also that metrics could be used to define the characteristics and number of workers to scale. currently we are processing 30 tables and each one must be generated for 800 tenant so it would be the execution of 24000 notebooks.

[Noopur_Nigam]

Hi @alexander grajales vanegas​ Are you creating all the databases and tables in gold zone manually?If so, please check out DLT https://docs.databricks.com/data-engineering/delta-live-tables/index.html, it will take care of your complete pipeline by itself.

[alexgv12]

Hi @Noopur Nigam​ , currently the creation is an automatic process that we implement in pyspark and using jinja to build the different zones, but it is very interesting this alternative that you show me, I will review the documentation to see what we could use or take it into account for future projects, thank you very much.