Hello, currently we have a process that builds with delta table the zones of bronze, silver and when it reaches gold we must create specific zones for each client because the schema changes, for this we create databases and separate tables, but when we are doing this process it takes a long time and in many occasions communication is lost in the notebook and the execution fails, what good practice, or alternatives could we do to improve our flow performance and also that metrics could be used to define the characteristics and number of workers to scale. currently we are processing 30 tables and each one must be generated for 800 tenant so it would be the execution of 24000 notebooks.
