Error running 80 task at same time in Job, how limit this?

Maxi1693 — Fri, 23 Feb 2024 15:46:15 GMT

Hi! I have a Job running to process multiple streaming tables.

In the beginning, it was working fine, but now I have 80 tables running in this job, the problem is that all the runs are trying to run at the same time throwing an error. Is there a way to limit the number of tasks that a job can execute per each run?

I have configured the Job cluster with autoscaling from 2 to 4. I thought it worked as a limitation to run per each of the 4 tasks, but I was wrong.

The error I am getting is "Failure starting repl. Try detaching and re-attaching the notebook.", as I could find it is because the cluster is overloaded, but I can not limit the number of runs in parallel.

Re: Error running 80 task at same time in Job, how limit this?

thauck — Wed, 04 Sep 2024 09:19:00 GMT

@Retired_mod wrote:
Additionally, configure the maximum number of concurrent tasks per worker node. You can set this in the cluster configuration under “Max Concurrency” or “Max Tasks”. Adjust this value based on your workload and available resources.

Hi. Can you please describe more detailed where we can find this configuration of maximum number of concurrent tasks? I couldn't find anything like this. We use Databricks Asset Bundles to define our job cluster.

topic Error running 80 task at same time in Job, how limit this? in Administration & Architecture

Error running 80 task at same time in Job, how limit this?

Re: Error running 80 task at same time in Job, how limit this?