cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Workflow timeout

Hubert-Dudek
Esteemed Contributor III

Always set a timeout for your jobs! It not only safeguards against unforeseen hang-ups but also optimizes resource utilization. Equally essential is to consider having a threshold warning. This can alert you before a potential failure, allowing proactive measures. And don’t forget to enable notifications both for threshold warnings and failed jobs.

ezgif-2-283506cee0.gif

4 REPLIES 4

jose_gonzalez
Databricks Employee
Databricks Employee

Thank you for sharing this @Hubert-Dudek 

sparkplug
New Contributor II

Hi
How do I set this as default for all users in my workspace. i.e. they shouldn't be able to run jobs for more than 24 hours

 

Panda
Valued Contributor

@sparkplug - You can achieve this in the cluster policy by setting the job timeout_seconds to 86400.

sparkplug
New Contributor II

We already have a policy and users are using clusters created with those to run their jobs. Since the policies are not based on job compute but on Power user compute, I am not able to set the job timeout_seconds.

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group