Multiple Databricks Issues: Spark Context Limit, Concurrency Load, API Character Limit & Job Timeout

[adhi_databricks]

I am encountering multiple issues in our Databricks environment and would appreciate guidance or best-practice recommendations for each. Details below:

1. [MaxSparkContextsExceeded] Too many execution contexts are open right now (Limit 150)

Error:

[MaxSparkContextsExceeded] Too many execution contexts are open right now. (Limit set currently to 150) Local : heap memory

• Suspecting that Spark contexts are not being released properly.

• Multiple scheduled notebooks may be causing accumulation.

Questions:

• Common causes of hitting this 150 SparkContext limit?

• How to inspect which jobs/notebooks are holding open contexts?

• Any cleanup patterns or cluster settings recommended?

2. 20 Concurrent Databricks Notebooks Triggered

We trigger ~20 notebooks at the same time on the same cluster.

Questions:

• Any Databricks concurrency limits at the cluster/job level?

• How to throttle or queue notebook runs?

3. Databricks API 10k Character Limit

We’re hitting a request size restriction (~10,000 characters) when interacting with Databricks API.

Questions:

• What is the official request/response size limit?

• Is the 10k cap configurable?

Request

Looking for:

• Explanation of why these happen

• How to diagnose root causes

• Recommended best practices for preventing them

Any guidance or references to Databricks documentation would be very helpful.

[K_Anudeep]

Hello @adhi_databricks ,

Good Day! Below are the answers to your questions:

 [MaxSparkContextsExceeded] Too many execution contexts are open right now (Limit 150)

• This issue occurs when many Spark execution context objects are open without being closed.
• Databricks creates an execution context each time a notebook attaches to a cluster; a cluster supports up to 150 contexts total (145 user REPLs + 5 internal). If many notebooks (especially scheduled ones) reuse the same long-lived cluster, or if idle contexts are not being evicted, you eventually hit this limit, and new runs fail with Too many execution contexts are open right now (Limit set currently to 150). Doc Link: https://kb.databricks.com/clusters/too-many-execution-contexts-are-open-right-now
• You can refer to the above document for best practices to avoid this issue in the future. As a best practice, use job clusters to avoid these issues in future.

20 Concurrent Databricks Notebooks Triggered

• The error is not about the no of notebooks you are using, but with the no of contexts being created on a single cluster.
• At the workspace level, Databricks supports up to 2,000 concurrently running tasks/jobs per workspace. Therefore, the recommended pattern is to fan out across multiple job clusters, rather than concentrating all notebook runs on a single shared cluster. Doc Link: https://docs.databricks.com/aws/en/resources/limits
• Concurrency control is performed at the job level, not via Spark contexts. The Jobs API and UI expose a max_concurrent_runs setting that limits the number of parallel runs of the same job (0 = effectively queue all new runs, 1 = fully serialised, up to 1000). Additional triggers beyond that limit are automatically queued by Databricks until a slot becomes free. Doc Link: https://docs.databricks.com/aws/en/reference/jobs-2.0-api#request-structure
• The general best practice is to define notebooks as tasks in Jobs, set max_concurrent_runs to a value that matches your cluster capacity, and optionally use multiple job clusters if you need high total throughput without overloading a single cluster.

Databricks API 10k Character Limit

• The limits in API's are in MB ,not characters
• For example, Jobs API 2.0 states that “the maximum allowed size of a request to the Jobs API is 10 MB”, and the SQL Statement Execution API caps the SQL text at 16 MiB with result-size limits. Doc Link: https://docs.databricks.com/aws/en/reference/jobs-2.0-api

[siva-anantha]

I would like to add my experience with 3. Databricks API 10k Character Limit

We had a similar issue, and this limit cannot be changed. Instead review concepts of sharing the input/output between Databricks and caller using cloud storage like ADLS. Provide ADLS URLs as input and output; this way we are not limited by size of payload.