- 10211 Views
- 2 replies
- 12 kudos
How concurrent runs in a job matches to cluster configuration?
In databricks jobs, there's a field to add concurrent runs which can be set to 1000.If I've a cluster with 4 worker nodes and 8 cores each, then at max how many concurrent jobs I'll be able to execute?What will happen if I launch 100 instances of sam...
- 10211 Views
- 2 replies
- 12 kudos
- 12 kudos
@Aman Sehgal​ On E2 workspace the limit is 1000 concurrent runs. If you trigger 100 runs​ at the same time, 100 clusters will be created and the runs will be executed. If you use the same cluster for 100 runs, then you might face a lot of failed jobs...
- 12 kudos
- 9977 Views
- 10 replies
- 2 kudos
Resolved! Max job concurrency per workspace
Per the documentation, a workspace is limited to 1k concurrent job runs.Can somebody clarify how the concurrency limit is set i.e:Is it 1k concurrent runs across all jobs in the workspaceIs it 1k concurrent runs for a single jobAlso, is there any way...
- 9977 Views
- 10 replies
- 2 kudos
- 2 kudos
Hi @Kaniz Fatma​ Weners gave some great suggestions on the issue we are dealing with. But some confirmation on the question from data bricks side would be much appreciated
- 2 kudos