Hello @adhi_databricks ,

Good Day! Below are the answers to your questions:

 [MaxSparkContextsExceeded] Too many execution contexts are open right now (Limit 150)

• This issue occurs when many Spark execution context objects are open without being closed.
• Databricks creates an execution context each time a notebook attaches to a cluster; a cluster supports up to 150 contexts total (145 user REPLs + 5 internal). If many notebooks (especially scheduled ones) reuse the same long-lived cluster, or if idle contexts are not being evicted, you eventually hit this limit, and new runs fail with Too many execution contexts are open right now (Limit set currently to 150). Doc Link: https://kb.databricks.com/clusters/too-many-execution-contexts-are-open-right-now
• You can refer to the above document for best practices to avoid this issue in the future. As a best practice, use job clusters to avoid these issues in future.

20 Concurrent Databricks Notebooks Triggered

• The error is not about the no of notebooks you are using, but with the no of contexts being created on a single cluster.
• At the workspace level, Databricks supports up to 2,000 concurrently running tasks/jobs per workspace. Therefore, the recommended pattern is to fan out across multiple job clusters, rather than concentrating all notebook runs on a single shared cluster. Doc Link: https://docs.databricks.com/aws/en/resources/limits
• Concurrency control is performed at the job level, not via Spark contexts. The Jobs API and UI expose a max_concurrent_runs setting that limits the number of parallel runs of the same job (0 = effectively queue all new runs, 1 = fully serialised, up to 1000). Additional triggers beyond that limit are automatically queued by Databricks until a slot becomes free. Doc Link: https://docs.databricks.com/aws/en/reference/jobs-2.0-api#request-structure
• The general best practice is to define notebooks as tasks in Jobs, set max_concurrent_runs to a value that matches your cluster capacity, and optionally use multiple job clusters if you need high total throughput without overloading a single cluster.

Databricks API 10k Character Limit

• The limits in API's are in MB ,not characters
• For example, Jobs API 2.0 states that “the maximum allowed size of a request to the Jobs API is 10 MB”, and the SQL Statement Execution API caps the SQL text at 16 MiB with result-size limits. Doc Link: https://docs.databricks.com/aws/en/reference/jobs-2.0-api