topic Re: Configure jobs throttling for ephemeral cluster ETLs in Data Engineering

Configure jobs throttling for ephemeral cluster ETLs

RicksDB — Thu, 30 Dec 2021 14:28:44 GMT

Hi,

Is it possible to configure job throttling in order to queue jobs across a workspace after a given number of concurrent execution when using the ephemeral cluster pattern? The reason is mainly for cost control. We prefer reducing performance rather than increasing cost if too many jobs are executed for various reasons.

Thanks

Re: Configure jobs throttling for ephemeral cluster ETLs

Anonymous — Thu, 30 Dec 2021 17:28:11 GMT

Hello @E H - Welcome and thank you for asking. My name is Piper, and I'm a moderator for Databricks. Let's see what the other members have to say. If we don't hear anything, we'll swing back to this.

Re: Configure jobs throttling for ephemeral cluster ETLs

Anonymous — Fri, 31 Dec 2021 00:18:02 GMT

If you have a fixed size cluster, this will happen automatically. Just don't turn on autoscaling.

https://docs.databricks.com/clusters/configure.html#cluster-size-and-autoscaling

Re: Configure jobs throttling for ephemeral cluster ETLs

RicksDB — Fri, 31 Dec 2021 00:34:51 GMT

Thanks for the answer josephk.

However, this solution doesn't work in my case.

If I launch 20 different jobs, I will have 20 ephemeral clusters running at the same time. Hence, if they each run for 5 mins, we will incur a bill of 100 min execution.

The idea would be, for example, to have a maximum configurable execution time for each 30 min. Jobs would be queued afterwards. In my example, I could just take my configured execution time * 48 and it would give me the worst possible case in a single day.

I also tried a scenario using a pool with a maximum set of VMs. However, instead of queuing jobs, additional jobs failed since new VMs couldn't be provisioned.

The objective is to be able to predict cost by calculating the worst case in a single day and ensure that we don't go beyond that. Right now, the only way I can do this is using an interactive cluster (and paying more DBU) instead of a job cluster.

Re: Configure jobs throttling for ephemeral cluster ETLs

Anonymous — Fri, 31 Dec 2021 00:38:50 GMT

Why not just run all the jobs on the same cluster? That will save you a lot of time not starting up 20 clusters.

Re: Configure jobs throttling for ephemeral cluster ETLs

RicksDB — Fri, 31 Dec 2021 00:42:00 GMT

That is only possible with an interactive cluster (cost more DBU). At least as far as I know.

Re: Configure jobs throttling for ephemeral cluster ETLs

Anonymous — Fri, 31 Dec 2021 00:49:15 GMT

Yes, that's correct. There is a new feature in the roadmap to reuse the same cluster which should help/speed things up.

Might still be worth it to do it all on 1 interactive cluster, which again shouldn't be too expensive for a smaller cluster with single node.

Re: Configure jobs throttling for ephemeral cluster ETLs

RicksDB — Fri, 31 Dec 2021 01:03:27 GMT

Thanks for the help josephk. I will continue to use an interactive cluster for the time being until the release of that new feature. Hopefully, it will allow my use case. Is there visibility on the roadmap for an ETA or more information on it?

Re: Configure jobs throttling for ephemeral cluster ETLs

Anonymous — Sun, 02 Jan 2022 00:00:16 GMT

The image is from the roadmap that was released in November, so it should be in preview sometime this month if it isn't already. Talk to your CSE about the preview testing.

Re: Configure jobs throttling for ephemeral cluster ETLs

BilalAslamDbrx — Mon, 03 Jan 2022 14:50:10 GMT

@E H that feature is in preview! Hit me up at bilal dot aslam at databricks dot com and I will get you enrolled in it.