Pass the job even if specific task fails

Puru20
New Contributor III

Hi , I have multiple data pipelines and each has data quality check as a final task which runs on dbt. There are 1500 test cases altogether runs everyday which is being captured on dashboard. Is there a way to pass the job even if this particular talk fail due to test failure as we can check dashboard for any test failures and do not want the workflow to fail. 

szymon_dybczak
Esteemed Contributor III

Hi @Puru20 ,

Yes, there is 🙂 Databricks determines whether a job run was successful based on the outcome of the job's leaf tasks. A leaf task is a task that has no downstream dependencies. 

So, just add another task that will have a dependency on your final task (so, now this task becomes your final one) and in that task you can define your success/failure logic. 

https://docs.databricks.com/aws/en/jobs/monitor#how-does-databricks-determine-job-run-status

View solution in original post

Puru20
New Contributor III

Hi @szymon_dybczak  The solution works perfectly when I set leaf job to pass irrespective of dbt test task status. Thanks much!

szymon_dybczak
Esteemed Contributor III

Hi @Puru20 , great that you managed to work it out. If the answer was helpful to you, please consider marking it as a best answer.