cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How many notebooks/jobs can I run in parallel on a Databricks cluster?

User16869510359
Esteemed Contributor

Is there a limit on it and is the limit configurable?

1 ACCEPTED SOLUTION

Accepted Solutions

User16869510359
Esteemed Contributor

There is a hard limit of 145 active execution contexts on a Cluster. This is to ensure the cluster is not overloaded with too many parallel threads starving for resources.

The limit is not configurable. If there are more than 145 parallel jobs to be executed, its strongly recommended to create a new cluster

View solution in original post

3 REPLIES 3

User16869510359
Esteemed Contributor

There is a hard limit of 145 active execution contexts on a Cluster. This is to ensure the cluster is not overloaded with too many parallel threads starving for resources.

The limit is not configurable. If there are more than 145 parallel jobs to be executed, its strongly recommended to create a new cluster

I would like to know where I can find the number of active execution contexts on a cluster, Can you please suggest?

Is it depend on the cluster size? If yes, what is the appropriate cluster configuration we are talking here.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.