cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

DLT maintenance job got stuck

tomnguyen_195
New Contributor III

Hi all,

Recently we just realize a huge cost associate with our databricks account and the main culprit of it is DLT's pipeline maintenance job that got auto-scheduled to run but got stucked and cost us thousand of DBU. Do you know what would be the reason for these jobs to get stuck randomly and how to avoid this huge cost?

Thanks,

Tom

2 REPLIES 2

tinai_long
New Contributor III

Same question. These maintenance jobs run for the maximum timeout (168 hours) and do not terminate. Example below:

Screen Shot 2022-06-30 at 11.39.44 PM

do you see any issues? Could you share the full error stack trace?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group