topic Re: Job cluster vs All purpose cluster in Data Engineering

Job cluster vs All purpose cluster

AmanSehgal — Fri, 16 Dec 2022 03:03:08 GMT

Environment: Azure

I've a workflow that takes approximately a minute to execute and I want to run the job every 2 minutes..

All purpose cluster:

On attaching all purpose cluster to the job, it takes approx. 60 seconds to execute.

Using job cluster:

On attaching job cluster to the job, it takes extra 30-45 seconds in `Pending` state, waiting for resource allocation in each job run.

What can be done to avoid job cluster spend that extra time to allocate resources?

The reason I want to use job cluster is because it costs $0.15/DBU and all purpose compute costs $0.40/DBU.

Re: Job cluster vs All purpose cluster

NhatHoang — Fri, 16 Dec 2022 04:22:56 GMT

Hi,

I am facing the same issue and I choose All Purpose cluster.

In my case, I need some jobs to run every 10 minutes.

- With All Purpose cluster, it take 1 minute

- With Job cluster, it takes 5 minutes (4 minutes for setting up + 1 minutes to perform task)

After compare price, I choose All Purpose. When this cluster is idle, I can use it for development or for other Notebook Dashboard.

🙂

Re: Job cluster vs All purpose cluster

AmanSehgal — Fri, 16 Dec 2022 06:35:12 GMT

@Hubert Dudek have you faced this challenge?

@Kaniz Fatma could you please help us resolve this with an SME. Thank you

Re: Job cluster vs All purpose cluster

Ajay-Pandey — Fri, 16 Dec 2022 06:50:42 GMT

Have you raised a request to databricks for this ?

Re: Job cluster vs All purpose cluster

Aviral-Bhardwaj — Fri, 16 Dec 2022 15:31:43 GMT

Hey @Aman Sehgal

This is a normal scenario all purpose cluster will complete your task in mere minutes of time but the job cluster has to setup itself up in the first place, but if you really want to reduce time, you can use the pool cluster option there.

Thanks

Aviral Bhardwaj

Re: Job cluster vs All purpose cluster

Priyag1 — Tue, 02 May 2023 18:27:54 GMT

Thanks for sharing

Re: Job cluster vs All purpose cluster

Priyag1 — Tue, 02 May 2023 18:28:24 GMT

Not neccesary