topic Re: Pausing a scheduled Azure Databricks job after failure in Data Engineering

Pausing a scheduled Azure Databricks job after failure

Dipesh — Tue, 31 Jan 2023 14:27:07 GMT

Hi All,

I have a job/workflow scheduled in Databricks to run after every hour.

How can I configure my Job to pause whenever a job run fails? (Pause the job/workflow on first failure)

I would want to prevent triggering multiple runs due to the scheduled/un-paused state of the job after the first failure and resume the schedule after the issue is fixed.

Thank you.

Re: Pausing a scheduled Azure Databricks job after failure

shan_chandra — Tue, 31 Jan 2023 17:41:52 GMT

@Dipesh Yogi - Please refer to the current behavior.

when you schedule workflow dependencies and configure each job has a task with a dependency of task2 to start until task1 complete. subsequent runs will not be triggered with the below message

Task <Task-name> failed. This caused all downstream tasks to get skipped.

Reference - https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/jobs#--task-dependencies.

The below documentation also explains the Repair and rerun feature of the workflows which address your specific scenario but only at the individual run level.

https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/how-to-fix-job-failures

Unfortunately, There is no mechanism currently to pause the workflow schedules after the first failure. However, you can create alerts to your email on the failure and upon receiving the alerts, manually stop the schedule. we will work internally on this new feature request to pause the schedule and it will be picked up based on the prioritization. Thanks for bringing this up!!!

Re: Pausing a scheduled Azure Databricks job after failure

Hubert-Dudek — Tue, 31 Jan 2023 19:16:14 GMT

You can pause a job using jobs Rest API. Just call it from the notebook when you catch the exception. https://<databricks-instance>/api/2.1/jobs/update

{
   "job_id":11223344,
   "new_settings":{
      "schedule":{
         "pause_status":"PAUSED"
      }
   }
}

more info here https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsUpdate

Re: Pausing a scheduled Azure Databricks job after failure

Dipesh — Wed, 01 Feb 2023 04:50:16 GMT

@Shanmugavel Chandrakasu Thank you for your response. We have enabled Databricks alerts but would be a problem during weekends and holidays 🙂

Also the data get updated after each runs so repairing the run after we detect the failure would lead to us losing some data.

Looking forward for this new feature.

Thanks again.🙏

Re: Pausing a scheduled Azure Databricks job after failure

Dipesh — Wed, 01 Feb 2023 04:53:45 GMT

Hi @Hubert Dudek , Thank you for your suggestion.

I understand that we can use Jobs API to change the pasue_status of job on errors, but sometimes we observed that the workflow/job fails due to cluster issues (while the job clusters are getting created) and before any of our code gets executed. In such scenarios I was wondering if there is any way to automatically pause the job.