Automating the re run of job (with several Tasks) // automate the notification of a failed specific tasks after re trying // Error handling on azure data factory pipeline with DataBricks notebook

Diego_MSFT — Sat, 06 Aug 2022 01:02:47 GMT

Hi DataBricks Experts:

I'm using Databricks on Azure.... I'd like to understand the following:

1) if there is way of automating the re run some specific failed tasks from a job (with several Tasks), for example if I have 4 tasks, and the task 1 and 2 have succeed and task 3 and 4 have failed, then to be able to re run task 3 and 4 one more time... I know there is a functionality per se inside Jobs, that allows to re run failed tasks but this needs to be done manually; and I want to automate this... here more info: https://docs.databricks.com/data-engineering/jobs/jobs.html (Repair an unsuccessful job run)

2) and if after retrying and still failing some tasks... then do some kind of notification invoking a https url for deciding what to do next after retrying 2 times. I've seen this documentation (about retrying several times on a notebook): https://docs.databricks.com/notebooks/notebook-workflows.html?_ga=2.96427868.191663080.1659650759-1341674114.1658447300.

I've seen this reference and it seems only email is allowed as notification method for failed job: https://stackoverflow.com/questions/61586505/azure-databricks-job-notification-email ... has this changed?

Additionally:

3) not sure if there is a best practice for orchestrating notebooks from Azure Data Factory and manage this type of problems from Azure Data Factory? I've seen this documentation: https://azure.microsoft.com/es-mx/blog/operationalize-azure-databricks-notebooks-using-data-factory/

Where it seems that if a notebook failed then this can be catch up on DataFactory pipeline, and manage the error there (for example sending an email).

Any help will be appreciated.

Thanks

Diego

Re: Automating the re run of job (with several Tasks) // automate the notification of a failed specific tasks after re trying // Error handling on azure data factory pipeline with DataBricks notebook

Lindberg — Thu, 20 Apr 2023 18:55:30 GMT

You can use "retries".

In Workflow, select your job, the task, and in the options below, configure retries.

If so, you can also see more options at:

https://learn.microsoft.com/pt-br/azure/databricks/dev-tools/api/2.0/jobs?source=recommendations

topic Automating the re run of job (with several Tasks) // automate the notification of a failed specific tasks after re trying // Error handling on azure data factory pipeline with DataBricks notebook in Data Engineering

Automating the re run of job (with several Tasks) // automate the notification of a failed specific tasks after re trying // Error handling on azure data factory pipeline with DataBricks notebook

Re: Automating the re run of job (with several Tasks) // automate the notification of a failed specific tasks after re trying // Error handling on azure data factory pipeline with DataBricks notebook