Job cluster vs All purpose cluster

[AmanSehgal]

Environment: Azure

I've a workflow that takes approximately a minute to execute and I want to run the job every 2 minutes..

All purpose cluster:

On attaching all purpose cluster to the job, it takes approx. 60 seconds to execute.

Using job cluster:

On attaching job cluster to the job, it takes extra 30-45 seconds in `Pending` state, waiting for resource allocation in each job run.

What can be done to avoid job cluster spend that extra time to allocate resources?

The reason I want to use job cluster is because it costs $0.15/DBU and all purpose compute costs $0.40/DBU.

[NhatHoang]

Hi,

I am facing the same issue and I choose All Purpose cluster.

In my case, I need some jobs to run every 10 minutes.

- With All Purpose cluster, it take 1 minute

- With Job cluster, it takes 5 minutes (4 minutes for setting up + 1 minutes to perform task)

After compare price, I choose All Purpose. When this cluster is idle, I can use it for development or for other Notebook Dashboard.

🙂

[AmanSehgal]

@Hubert Dudek​ have you faced this challenge?

@Kaniz Fatma​ could you please help us resolve this with an SME. Thank you

[Ajay-Pandey]

Have you raised a request to databricks for this ?

[Priyag1]

Not neccesary

[Aviral-Bhardwaj]

Hey @Aman Sehgal​ 

This is a normal scenario all purpose cluster will complete your task in mere minutes of time but the job cluster has to setup itself up in the first place, but if you really want to reduce time, you can use the pool cluster option there.

Thanks

Aviral Bhardwaj

[Priyag1]

Thanks for sharing