DLT: Waiting for resources took a long time

[Gilg]

Hi Team,

I have a DLT pipeline running in Production for quite some time now. When I check the pipeline, a couple of jobs took longer than expected. Usually, 1 job only took 10-15 minutes to complete with 2 to 3 mins to provision a resource. Then I have couple of jobs that failed after it reaches 30 minutes of trying to provision a resource.

What could not sure what happened here.

Cheers,

G

[Kaniz]

Hi @Gilg, The issue you're experiencing with your DLT pipeline could be due to a couple of factors:

1. **Development Optimizations**: As per the Databricks release notes from September 7-13, 2021, new pipelines run in development mode by default. This mode optimizes pipeline execution by reusing clusters and turning off automatic retries. If your pipeline was created after this change and runs in development mode, reusing clusters could cause longer provisioning times if the clusters are busy with other tasks.   

2. **Schema Changes**: According to the release notes from November 30 - December 6, 2021, if your pipeline reads data with Auto Loader and the schema of the input data changes while an update is running, the update is logged as CANCELED and automatically retried. This could cause longer job times and even job failures if the schema changes are significant or frequent.   

3. **Bug Fixes**: A bug caused the DLT UI to show a pipeline in a RUNNING state even after update failure. This bug has been fixed in a recent release. If your pipeline was affected by this bug, it could explain the job failures you're seeing.   

You may want to consider these factors when troubleshooting your pipeline. If none of these are the cause, you might want to contact Databricks support for further assistance.