Tasks in job are in pending state

alesventus
Contributor

‎07-04-2024 12:38 AM

I have databricks job with around 70 notebooks. When job starts, only one notebook gets executed and the rest of the notebooks that are at the beginning of the line are in state PENDING (not blocked). Looks like notebooks cannot run in parallel for the first 5 minutes. Then start all other notebooks that are first in the lines and job runs in parallel.

This behavior started few days ago and before when job started, all notebooks got executed from left to right.

Any idea why is this happening or how to find the reason for this?

Thanks

job_start.jpg

job_middle.jpg

  

0 Kudos

szymon_dybczak
Esteemed Contributor III

‎07-04-2024 01:51 AM

Maybe something related to autoscaling options? So when databricks detects increased workload it will scale up number of workers and then the rest of notebooks get executed. Do you use DLT ?

0 Kudos

alesventus
Contributor
In response to szymon_dybczak

‎07-04-2024 03:28 AM

Autoscaling is turned off because I set up manually 5 workers. DLT is not used, these notebooks just merge new data from the source to the bronze zone

0 Kudos