Databricks cluster launch time

[Farzana]

Hi Team,

We have an @adf pipeline which will run some set of activities before #Azure databricks notebooks get called.As and when the notebooks are called our pipeline will launch a new cluster for every job with job compute as Standard F4 with a single worker node.To launch the cluster itself it is taking ~7mins which increases the overall ADF pipeline run time.

Could you please suggest a solution to reduce the cluster launch time?

Note:Our ADF pipeline has an event based trigger which will run as and when there is a file comes to ADLS. We cannot have a cluster created and running all the time as it impacts the cost.

Thanks

[Kaniz_Fatma]

Hi @Farzana , If the cluster pool has enough idle instances and the slow image download from the artefact storage is not the issue, then a few things can be done to reduce the cluster launch time.

• Preload the runtime on the instance pool to improve the problem.
• Investigate the node daemon logs to identify any issues causing the delay.
• Check for any custom spark configurations and libraries that might be causing the issue.

[Farzana]

@Kaniz_Fatma Thanks for the response. Could you please elaborate what do you mean by preloading the runtime on instance pool?

Even the cluster pool needs to run continuously(as there is no specific time period defined for the files to come to ADLS) in order to reduce the launch time of cluster for each and every job so that the over all ADF pipeline run time can be fast.isn't it?

Please help me in understanding "preload the runtime on the instance pool"

Thanks

[Kaniz_Fatma]

Hi @Farzana, Preloading the runtime on the instance pool is a feature that allows selecting a specific version of the runtime to be loaded on idle instances in the pool. This can speed up pool-backed cluster launches and reduce cluster start and auto-scaling times. By selecting a preloaded Spark version, a cluster can launch even more quickly than a pool-backed cluster that doesn't use a preloaded version. This feature can potentially improve the slow cluster start times that the customer is experiencing. The feature is available on Databricks as Fleet Instance.

Source:
- https://docs.databricks.com/release-notes/product/2019/july.html
- https://docs.databricks.com/clusters/pools.html
- https://docs.databricks.com/compute/aws-fleet-instances.html