How could we automatically re run the complete job if any of its associted task fails.?

[Milliman]

I need to re run the compete job automatically if any of its associated task gets failed, any help would be appreciable. Thanks

[Kaniz_Fatma]

Hi @Milliman, In Databricks, you can automate the re-run of a job if any of its associated tasks fail.

Here are some steps to achieve this:

1. Conditional Task Execution:

   • You can specify “Run if dependencies” to run a task based on the run status of its dependencies. For example, even if some or all of the dependencies have failed, using this option allows your job ...¹².
   • To set this up:
     • In your Databricks job, select the specific task.
     • Configure the retries option in the task settings. This allows you to retry the task automatically if it fails³.
     • You can also explore additional options for retries in the Databricks API documentation³.

2. Error Handling and Repair:

   • After identifying the cause of failure, you can repair failed or canceled multi-task jobs by running only the subset of unsuccessful tasks and their dependent tasks. Successful tasks and their dependents are not re-run, which reduces the time and resources needed fo...⁴.
   • This feature allows you to selectively re-run specific tasks that failed, rather than the entire job.

3. Notifications:

   • While Databricks currently supports email notifications for failed jobs, you can also consider invoking an HTTP URL for further decision-making after retrying tasks.
   • If you need more customized notifications, you might need to handle them externally (e.g., using Azu...³.

If you encounter any issues or need further assistance, feel free to ask! 🚀