Error running 80 task at same time in Job, how limit this?

[Maxi1693]

Hi! I have a Job running to process multiple streaming tables. 

In the beginning, it was working fine, but now I have 80 tables running in this job, the problem is that all the runs are trying to run at the same time throwing an error. Is there a way to limit the number of tasks that a job can execute per each run?

I have configured the Job cluster with autoscaling from 2 to 4. I thought it worked as a limitation to run per each of the 4 tasks, but I was wrong.

The error I am getting is "Failure starting repl. Try detaching and re-attaching the notebook.", as I could find it is because the cluster is overloaded, but I can not limit the number of runs in parallel. 

[Kaniz]

Hi @Maxi1693, It appears that you’re encountering issues with parallel execution of tasks in your Databricks job.

Let’s address this by considering a few strategies:

1. Concurrency Limit for Tasks:

   • Databricks allows a maximum of 1000 concurrent task runs within a workspace. If you’re hitting this limit, you’ll receive a 429 Too Many Requests response when requesting runs t...¹².
   • To mitigate this, consider staggering the execution of your streaming tables. You can divide them into smaller batches and schedule them at different intervals to avoid overwhelming the cluster.

2. Job Creation Rate:

   • The number of jobs a workspace can create in an hour is limited to 10,000 (including “runs submit”). This limit also affects jobs created via the REST API and notebook workflows¹².
   • If you’re creating jobs frequently, you might be hitting this hourly limit. Consider optimizing your job creation process or spreading out job submissions.

3. Autoscaling and Cluster Configuration:

   • While autoscaling helps manage cluster resources dynamically, it doesn’t inherently limit the number of concurrent runs.
   • Instead of relying solely on autoscaling, consider setting a fixed cluster size (e.g., 4 workers) to ensure consistent resources for your tasks.
   • Additionally, configure the maximum number of concurrent tasks per worker node. You can set this in the cluster configuration under “Max Concurrency” or “Max Tasks”. Adjust this value based on your workload and available resources.

4. Notebook Execution Order:

   • If your streaming tables are processed within notebooks, ensure that they execute sequentially rather than concurrently.
   • You can use notebook workflows or job dependencies to control the order of execution.

5. Cluster Overload and REPL Errors:

   • The “Failure starting repl” error often occurs when the cluster is overloaded due to excessive concurrent tasks.
   • Detaching and re-attaching the notebook might temporarily alleviate the issue, but it’s not a long-term solution.
   • Focus on managing concurrency and resource allocation to prevent cluster overload.

Remember to monitor your job execution patterns, adjust cluster settings, and stagger your tasks to maintain a balance between parallelism and resource availability.