DLT: Waiting for resources took a long time

[Gilg]

Hi Team,

I have a DLT pipeline running in Production for quite some time now. When I check the pipeline, a couple of jobs took longer than expected. Usually, 1 job only took 10-15 minutes to complete with 2 to 3 mins to provision a resource. Then I have couple of jobs that failed after it reaches 30 minutes of trying to provision a resource.

What could not sure what happened here.

Cheers,

G

[speaker_city]

I am currently trying projects from dbdemos [Full Delta Live Tables Pipeline - Loan].

I keep running into this error. how do I resolve this?

[alfredo594]

Hello,

To troubleshoot performance issues in your DLT pipeline, review pipeline logs, profile the pipeline, optimize data transformations, increase parallelism, and adjust resource allocation. Identify and address resource provisioning delays, data volume and complexity issues, and pipeline bottlenecks. By systematically addressing these potential causes, you can improve the performance of your DLT pipeline.