Facing issue with Databricks Context Limit Exceeded

[ganemouni]

We have a use case , where there may be chance of 200 jobs executing at once. But Few notebooks are failing with an issue "run failed with error message Too many execution contexts are open right now.(Limit set currently to 150)." Can anyone help how to solve this?

[Walter_C]

Hello, you can refer to the following KB article for information and best practices in regards the issue you are facing: https://kb.databricks.com/en_US/notebooks/too-many-execution-contexts-are-open-right-now 

Best practices

• Use a job cluster instead of an interactive cluster. A job cluster for each job is the best way to avoid running out of execution contexts. Job clusters should be used for isolation and reliability.
• Reduce the number of separate notebooks used to reduce the number of execution contexts required.