cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
AmanSehgal
by • Honored Contributor III
  • 5064 Views
  • 2 replies
  • 12 kudos

How concurrent runs in a job matches to cluster configuration?

In databricks jobs, there's a field to add concurrent runs which can be set to 1000.If I've a cluster with 4 worker nodes and 8 cores each, then at max how many concurrent jobs I'll be able to execute?What will happen if I launch 100 instances of sam...

Data Engineering
Reply
  • 5064 Views
  • 2 replies
  • 12 kudos
View Replies
Latest Reply
Prabakar
Esteemed Contributor III
  • 12 kudos

@Aman Sehgal​ On E2 workspace the limit is 1000 concurrent runs. If you trigger 100 runs​ at the same time, 100 clusters will be created and the runs will be executed. If you use the same cluster for 100 runs, then you might face a lot of failed jobs...

  • 12 kudos
1 More Replies
hari
by • Contributor
  • 6036 Views
  • 10 replies
  • 2 kudos

Resolved! Max job concurrency per workspace

Per the documentation, a workspace is limited to 1k concurrent job runs.Can somebody clarify how the concurrency limit is set i.e:Is it 1k concurrent runs across all jobs in the workspaceIs it 1k concurrent runs for a single jobAlso, is there any way...

Data Engineering
Reply
  • 6036 Views
  • 10 replies
  • 2 kudos
View Replies
Latest Reply
hari
Contributor
  • 2 kudos

Hi @Kaniz Fatma​ Weners gave some great suggestions on the issue we are dealing with. But some confirmation on the question from data bricks side would be much appreciated

  • 2 kudos
9 More Replies
Labels