https://community.databricks.com/t5/data-engineering/performance-issue-running-50-notebooks-from-adf/m-p/47860#M28213 <P>I have process in Data factory, that loads CDC changes from sql server and then trigger notebook with merge to bronze and silver zone. Single notebook takes about 1 minute to run but when all 50 notebooks are fired at once the whole process takes 25 minutes.&nbsp;</P><P>There is not a lot of changes in sql tables. When notebooks run, cluster must scale up and it takes much more time to finish.</P><P>Is it really a big deal for cluster to run 50 notebooks in parallel?</P><P>cluster config: 12.2 LTS access mode shared</P><P>Photon enabled</P><P>worker: 2-8 standard DS3 v2</P><P>driver: standard DS3 v2</P><P>here is screenshot from ganglia - load starts at 0600</P> Tue, 03 Oct 2023 13:28:06 GMT alesventus 2023-10-03T13:28:06Z https://community.databricks.com/t5/data-engineering/performance-issue-running-50-notebooks-from-adf/m-p/47860#M28213 <P>I have process in Data factory, that loads CDC changes from sql server and then trigger notebook with merge to bronze and silver zone. Single notebook takes about 1 minute to run but when all 50 notebooks are fired at once the whole process takes 25 minutes.&nbsp;</P><P>There is not a lot of changes in sql tables. When notebooks run, cluster must scale up and it takes much more time to finish.</P><P>Is it really a big deal for cluster to run 50 notebooks in parallel?</P><P>cluster config: 12.2 LTS access mode shared</P><P>Photon enabled</P><P>worker: 2-8 standard DS3 v2</P><P>driver: standard DS3 v2</P><P>here is screenshot from ganglia - load starts at 0600</P> Tue, 03 Oct 2023 13:28:06 GMT https://community.databricks.com/t5/data-engineering/performance-issue-running-50-notebooks-from-adf/m-p/47860#M28213 alesventus 2023-10-03T13:28:06Z