topic Re: delta table separate gold zone by different tenant in Data Engineering

delta table separate gold zone by different tenant

alexgv12 — Thu, 28 Apr 2022 21:34:24 GMT

Hello, currently we have a process that builds with delta table the zones of bronze, silver and when it reaches gold we must create specific zones for each client because the schema changes, for this we create databases and separate tables, but when we are doing this process it takes a long time and in many occasions communication is lost in the notebook and the execution fails, what good practice, or alternatives could we do to improve our flow performance and also that metrics could be used to define the characteristics and number of workers to scale. currently we are processing 30 tables and each one must be generated for 800 tenant so it would be the execution of 24000 notebooks.

Re: delta table separate gold zone by different tenant

Noopur_Nigam — Tue, 02 Aug 2022 05:20:33 GMT

Hi @alexander grajales vanegas Are you creating all the databases and tables in gold zone manually?If so, please check out DLT https://docs.databricks.com/data-engineering/delta-live-tables/index.html, it will take care of your complete pipeline by itself.

Re: delta table separate gold zone by different tenant

alexgv12 — Wed, 03 Aug 2022 02:56:25 GMT

Hi @Noopur Nigam , currently the creation is an automatic process that we implement in pyspark and using jinja to build the different zones, but it is very interesting this alternative that you show me, I will review the documentation to see what we could use or take it into account for future projects, thank you very much.