Delta Live Tables: Too much time to do the "setting up"

[jorgemarmol]

Hello community!

Recently I have been working in delta live table for a big project. My team and me have been studying so much and finally we have built a good pipeline with CDC that load 608 entities (and, therefore, 608 delta live tables and 608 materialized views).

The issue is not the load, although there are many tables, the issue is the setting up process. The total time is 1h 30 min in which almost 1 h (or more) is setting up the tables process. 

Anyone have an idea of why could have been happening it or how could I accelerate this first process? Thank you so much! 

[PearceR]

I have been having similar problems, any insight into this would be great.

[anandh]

I am facing same issues while acquiring the resource to run the DLT pipeline and its taking too much time to setup resource for DLT run.

[PearceR]

I had to scrap using DLT in the end for so many entities, the other option is to run in continuously but this is expensive.

[-werners-]

Interesting...
DLT probably spends x seconds/table for the setup.
If you have time, you could do some tests to see if the table setup scales linearly (1 table, 5 sec for setup, 10 tables 50 sec etc).
If you do, please share the outcome.