Hi Guys,
We have a DLT pipeline that is reading data from landing to raw (csv files into tables) for approximately 80 tables.
In our first attempt at this we declared each table separately in a python notebook. One @Dlt table declared per cell.
Then when another database came along with 300 tables we looked for a better solution and found a way to dynamically declare the dlt tables using a loop and a table with the table names we want to declare. This works good and now there is no repetition of code like before. However I discovered a trade off I hope we can get around.
Using the first method where the tables are statically declared on each line the INITAILIZATION and SETTING UP TABLES stages are taking only 4 minutes together. But when we use the dynamic declaration method its now taking 30 to 35 minutes for those same stages.
Has anyone else who is using DLT dynamic table declaration encountered this big jump in run-time using this style? A 30 minute jump in run time seems excessive to me just to produce the dlt declarations.
Again, thanks for any suggestions or help
