12-15-2022 07:03 PM
Environment: Azure
I've a workflow that takes approximately a minute to execute and I want to run the job every 2 minutes..
All purpose cluster:
On attaching all purpose cluster to the job, it takes approx. 60 seconds to execute.
Using job cluster:
On attaching job cluster to the job, it takes extra 30-45 seconds in `Pending` state, waiting for resource allocation in each job run.
What can be done to avoid job cluster spend that extra time to allocate resources?
The reason I want to use job cluster is because it costs $0.15/DBU and all purpose compute costs $0.40/DBU.
12-15-2022 08:22 PM
Hi,
I am facing the same issue and I choose All Purpose cluster.
In my case, I need some jobs to run every 10 minutes.
- With All Purpose cluster, it take 1 minute
- With Job cluster, it takes 5 minutes (4 minutes for setting up + 1 minutes to perform task)
After compare price, I choose All Purpose. When this cluster is idle, I can use it for development or for other Notebook Dashboard.
🙂
12-15-2022 10:35 PM
@Hubert Dudek have you faced this challenge?
@Kaniz Fatma could you please help us resolve this with an SME. Thank you
12-15-2022 10:50 PM
Have you raised a request to databricks for this ?
05-02-2023 11:28 AM
Not neccesary
12-16-2022 07:31 AM
Hey @Aman Sehgal
This is a normal scenario all purpose cluster will complete your task in mere minutes of time but the job cluster has to setup itself up in the first place, but if you really want to reduce time, you can use the pool cluster option there.
Thanks
Aviral Bhardwaj
05-02-2023 11:27 AM
Thanks for sharing
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.