cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT: Waiting for resources took a long time

Gilg
Contributor II

Hi Team,

I have a DLT pipeline running in Production for quite some time now. When I check the pipeline, a couple of jobs took longer than expected. Usually, 1 job only took 10-15 minutes to complete with 2 to 3 mins to provision a resource. Then I have couple of jobs that failed after it reaches 30 minutes of trying to provision a resource.

Gilg_0-1696540251644.png

What could not sure what happened here.

Cheers,

G

 

2 REPLIES 2

speaker_city
New Contributor II

I am currently trying projects from dbdemos [Full Delta Live Tables Pipeline - Loan].

I keep running into this error. how do I resolve this?

Screenshot 2024-09-03 at 7.14.29 PM.png

alfredo594
New Contributor II

Hello,

To troubleshoot performance issues in your DLT pipeline, review pipeline logs, profile the pipeline, optimize data transformations, increase parallelism, and adjust resource allocation. Identify and address resource provisioning delays, data volume and complexity issues, and pipeline bottlenecks. By systematically addressing these potential causes, you can improve the performance of your DLT pipeline.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group