Job cluster vs All purpose cluster
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-15-2022 07:03 PM
Environment: Azure
I've a workflow that takes approximately a minute to execute and I want to run the job every 2 minutes..
All purpose cluster:
On attaching all purpose cluster to the job, it takes approx. 60 seconds to execute.
Using job cluster:
On attaching job cluster to the job, it takes extra 30-45 seconds in `Pending` state, waiting for resource allocation in each job run.
What can be done to avoid job cluster spend that extra time to allocate resources?
The reason I want to use job cluster is because it costs $0.15/DBU and all purpose compute costs $0.40/DBU.
Labels:
- Labels:
-
Cluster
-
Dbu
-
Purpose Cluster
-
Workflow