Facing issue with Databricks Context Limit Exceeded

ganemouni
New Contributor

‎06-19-2024 06:17 AM

We have a use case , where there may be chance of 200 jobs executing at once. But Few notebooks are failing with an issue "run failed with error message Too many execution contexts are open right now.(Limit set currently to 150)." Can anyone help how to solve this?

ganemouni_0-1718803036914.png

 

0 Kudos

Walter_C
Databricks Employee
Databricks Employee

‎06-19-2024 12:04 PM

Hello, you can refer to the following KB article for information and best practices in regards the issue you are facing: https://kb.databricks.com/en_US/notebooks/too-many-execution-contexts-are-open-right-now 

Best practices

  • Use a job cluster instead of an interactive cluster. A job cluster for each job is the best way to avoid running out of execution contexts. Job clusters should be used for isolation and reliability.
  • Reduce the number of separate notebooks used to reduce the number of execution contexts required.
0 Kudos