Select job cluster vs all purpose cluster
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-11-2023 09:10 AM
I have workflow and need to run at every 1 minute interval, it is rest api call, should I go for all purpose cluster or job cluster to meet the SLA. We need to get the as soon as it is available.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-12-2023 12:03 AM
I’m bit confused.
when I scheduled the job in job cluster. It took nearly 3 mins to spin up back.
so that I can’t meet the SLA of 1 min.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-02-2024 05:58 AM
Thank you Kaniz_Fatma for these clear insights on All-purpose and Job clusters
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-12-2023 06:20 PM
@greyamber Interactive cluster costs two time more than job cluster. can you explain use-case of why job API needs to invoked and what API is doing.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-14-2023 08:32 AM
We need to pull data from an api endpoint at every 1 minute interval. Time to fetch and process is 30 seconds.
if we use job cluster once the process completes it get terminated and for next run we have to wait ~3-5 mins to spin up the cluster. Which over pass our SLA 1 minute.
