Pass the job even if specific task fails

[Puru20]

Hi , I have multiple data pipelines and each has data quality check as a final task which runs on dbt. There are 1500 test cases altogether runs everyday which is being captured on dashboard. Is there a way to pass the job even if this particular talk fail due to test failure as we can check dashboard for any test failures and do not want the workflow to fail. 

[szymon_dybczak · Accepted solution]

Hi @Puru20 ,

Yes, there is 🙂 Databricks determines whether a job run was successful based on the outcome of the job's leaf tasks. A leaf task is a task that has no downstream dependencies. 

So, just add another task that will have a dependency on your final task (so, now this task becomes your final one) and in that task you can define your success/failure logic. 

https://docs.databricks.com/aws/en/jobs/monitor#how-does-databricks-determine-job-run-status

[Puru20]

Hi @szymon_dybczak  The solution works perfectly when I set leaf job to pass irrespective of dbt test task status. Thanks much!

[szymon_dybczak]

Hi @Puru20 , great that you managed to work it out. If the answer was helpful to you, please consider marking it as a best answer.