topic Re: Select job cluster vs all purpose cluster in Data Engineering

Select job cluster vs all purpose cluster

greyamber — Mon, 11 Dec 2023 17:10:03 GMT

I have workflow and need to run at every 1 minute interval, it is rest api call, should I go for all purpose cluster or job cluster to meet the SLA. We need to get the as soon as it is available.

Re: Select job cluster vs all purpose cluster

greyamber — Tue, 12 Dec 2023 08:03:14 GMT

I’m bit confused.
when I scheduled the job in job cluster. It took nearly 3 mins to spin up back.
so that I can’t meet the SLA of 1 min.

Re: Select job cluster vs all purpose cluster

kulkpd — Wed, 13 Dec 2023 02:20:17 GMT

@greyamber Interactive cluster costs two time more than job cluster. can you explain use-case of why job API needs to invoked and what API is doing.

Re: Select job cluster vs all purpose cluster

greyamber — Thu, 14 Dec 2023 16:32:02 GMT

We need to pull data from an api endpoint at every 1 minute interval. Time to fetch and process is 30 seconds.
if we use job cluster once the process completes it get terminated and for next run we have to wait ~3-5 mins to spin up the cluster. Which over pass our SLA 1 minute.

Re: Select job cluster vs all purpose cluster

azde89 — Wed, 02 Oct 2024 12:58:46 GMT

Thank you Kaniz_Fatma for these clear insights on All-purpose and Job clusters