topic I have a job with strict performance requirements running in Azure Databricks. I had configured pools to reduce the startup time - but of late I'm seeing that the job cluster startup time is in the order of 5 to 8 minutes in Data Engineering

I have a job with strict performance requirements running in Azure Databricks. I had configured pools to reduce the startup time - but of late I'm seeing that the job cluster startup time is in the order of 5 to 8 minutes

sajith_appukutt — Tue, 08 Jun 2021 05:48:03 GMT

Despite having it configured to run from pools, my Azure Databricks jobs cluster is taking 5 to 8 minutes to launch. Any pointers ?

Re: I have a job with strict performance requirements running in Azure Databricks. I had configured pools to reduce the startup time - but of late I'm seeing that the job cluster startup time is in the order of 5 to 8 minutes

Ryan_Chynoweth — Fri, 18 Jun 2021 21:47:16 GMT

If you are using pools then you should consider keeping a min idle count of machines greater than 2. This will allow you to have machines available and ready to use. If you have 0 machines on idle then the first job executed against the pool will have to wait for the machines to become available.