topic Re: Tasks in job are in pending state in Data Engineering

Tasks in job are in pending state

alesventus — Thu, 04 Jul 2024 07:38:24 GMT

I have databricks job with around 70 notebooks. When job starts, only one notebook gets executed and the rest of the notebooks that are at the beginning of the line are in state PENDING (not blocked). Looks like notebooks cannot run in parallel for the first 5 minutes. Then start all other notebooks that are first in the lines and job runs in parallel.

This behavior started few days ago and before when job started, all notebooks got executed from left to right.

Any idea why is this happening or how to find the reason for this?

Thanks

Re: Tasks in job are in pending state

szymon_dybczak — Thu, 04 Jul 2024 08:51:48 GMT

Maybe something related to autoscaling options? So when databricks detects increased workload it will scale up number of workers and then the rest of notebooks get executed. Do you use DLT ?

Re: Tasks in job are in pending state

alesventus — Thu, 04 Jul 2024 10:28:54 GMT

Autoscaling is turned off because I set up manually 5 workers. DLT is not used, these notebooks just merge new data from the source to the bronze zone