Tasks in job are in pending state

[alesventus]

I have databricks job with around 70 notebooks. When job starts, only one notebook gets executed and the rest of the notebooks that are at the beginning of the line are in state PENDING (not blocked). Looks like notebooks cannot run in parallel for the first 5 minutes. Then start all other notebooks that are first in the lines and job runs in parallel.

This behavior started few days ago and before when job started, all notebooks got executed from left to right.

Any idea why is this happening or how to find the reason for this?

Thanks

  

[szymon_dybczak]

Maybe something related to autoscaling options? So when databricks detects increased workload it will scale up number of workers and then the rest of notebooks get executed. Do you use DLT ?

[alesventus]

Autoscaling is turned off because I set up manually 5 workers. DLT is not used, these notebooks just merge new data from the source to the bronze zone