Resource exhaustion when using default apply_changes python functionality

[lena1]

Hello!

We are currently setting up streaming  CDC pipelines for more than 500 tables. 

Due to the high number of tables, we split our tables into multiple pipelines, we use multiple DLT pipelines per layer: bronze, silver gold

In silver, we only upsert using the apply_changes function as describe here : https://docs.databricks.com/en/delta-live-tables/python-ref.html#cdc

However, we see that when we run silver pipelines, that there is an exhaustion in memory as well as the nodes which are used. 

So now, in order to reduce costs, we are wondering why this is the case and how to optimize it. It is a bit hard as the code here is not open source and we couldn't find any reference yet for optimizing the apply_changes function. 

Does anybody know what is happening in the background/ why it is taking that many resources and how to optimize it? 

Any help is welcome!

Thank you in advance! 

[Wojciech_BUK]

Hi Lena1,
there is no magic behind the scene.
If you write readstream from bronze table and writestream with ForEachBatch(function) and in function you will write MERGE stattemnt this will have similiar performance.
Maybe there is a lot of shuffeling happening or source tables are not optimized (a lot of small files).
This is hard to tell w/o looking on data and metrics 😞 
Wojciech