How concurrent runs in a job matches to cluster configuration?

[AmanSehgal]

In databricks jobs, there's a field to add concurrent runs which can be set to 1000.

If I've a cluster with 4 worker nodes and 8 cores each, then at max how many concurrent jobs I'll be able to execute?

What will happen if I launch 100 instances of same job at once over a cluster with 4 worker nodes (8 cores each)?

[Prabakar]

@Aman Sehgal​ On E2 workspace the limit is 1000 concurrent runs. If you trigger 100 runs​ at the same time, 100 clusters will be created and the runs will be executed. If you use the same cluster for 100 runs, then you might face a lot of failed jobs or even the cluster might go down as it will be overloaded.

[AmanSehgal]

Ok. So at max how many jobs can run concurrently on a cluster with 4 nodes and 8 cores each and 32gb memory.

Let's say incoming data in each file is 100kb and there are 100 such files arriving at once.

How can I do estimate calculations?