Hi, Im trying to find out what is bottleneck on cluster when running loading process.

Scenario: Loading CDC changes from sql server to Raw zone and merge changes into Bronze zone and then merge Bronze to Silver. All is orchestrated in data factory as a loop per table. Parallel run in data factory is set to 5. So max 5 tables and notebooks are running simultaneously. 

Strange is that when process starts, first 25 minutes cluster is not performing merges and first 5 notebooks take about 25 minutes. Then, cluster is autoscaled from 2 to 6 workers and the rest 45 notebooks finish in 10 minutes.

I tried to set 6 workers to cluster and start it with startup job 10 minutes before the load, but the result was the same.

I can see there is peak of memory use on driver from the beginning, workers memory or cpu dont peak. So I believe that bottleneck is driver trying to spread work across the workers. Am I correct?

I attached screenshots from ganglia. First is state 10 minutes after start, second is state after 30 minutes and third is after autoscaling 35 minutes after start.

Thanks.