DLT setup taking longer than actually building the tables

dsldan
Databricks Partner

Hi all!

We are using DLT for our ETL jobs, and we're noticing the setup steps (Initializing, Resetting tables. Setting up tables, Rendering graph) are taking much longer than actually ETL'ing the data into our tables. We have about 110 tables combined in our bronze, silver and gold layers. It seems using a bigger compute cluster does not help reduce this time.

Is there any way to speed this up?

Cheers,

Daan