topic Re: How concurrent runs in a job matches to cluster configuration? in Data Engineering

How concurrent runs in a job matches to cluster configuration?

AmanSehgal — Fri, 15 Jul 2022 02:46:33 GMT

In databricks jobs, there's a field to add concurrent runs which can be set to 1000.

If I've a cluster with 4 worker nodes and 8 cores each, then at max how many concurrent jobs I'll be able to execute?

What will happen if I launch 100 instances of same job at once over a cluster with 4 worker nodes (8 cores each)?

Re: How concurrent runs in a job matches to cluster configuration?

Prabakar — Sat, 16 Jul 2022 00:08:12 GMT

@Aman Sehgal On E2 workspace the limit is 1000 concurrent runs. If you trigger 100 runs at the same time, 100 clusters will be created and the runs will be executed. If you use the same cluster for 100 runs, then you might face a lot of failed jobs or even the cluster might go down as it will be overloaded.

Re: How concurrent runs in a job matches to cluster configuration?

AmanSehgal — Sat, 16 Jul 2022 02:56:05 GMT

Ok. So at max how many jobs can run concurrently on a cluster with 4 nodes and 8 cores each and 32gb memory.

Let's say incoming data in each file is 100kb and there are 100 such files arriving at once.

How can I do estimate calculations?