cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Workflows: Running dependent task despite earlier task fail

sharpbetty
New Contributor II

I have a scheduled task running in workflow.

Task 1 computes some parameters then these are picked up by a dependent reporting task: Task 2.

I want Task 2 to report "Failure" if Task 1 fails. Yet creating a dependency in workflows means that Task 2 will not run if Task 1 fails.

Any suggestions how I can keep the parameter sharing and dependency from Task 1 to Task 2, yet also allow Task 2 to fire even on failure of Task 1?

Edit: Have screenshot attached now, showing Task 2 skipped on a fail for Task 1.

2 REPLIES 2

Thanks @Retired_mod for your reply!

But I'm a little lost. I have found that if Task 1 fails, Task 2 is skipped, due to the dependency. (See screenshot). So there is no way I can see to execute Task 2 on a failure at Task 1.

I could disconnect the tasks (remove dependency), but my understanding is that if both tasks exist independently in the workflow, they will fire in parallel on schedule, meaning Task 2 doesn't wait for the parameters generated from Task 1.

I also thought of loading the tasks into different workflows, on a staggered schedule, but then I can't pass the parameters between them as they are in different flows.

NerdSan
New Contributor II

Hi @sharpbetty , 

Any suggestions how I can keep the parameter sharing and dependency from Task 1 to Task 2, yet also allow Task 2 to fire even on failure of Task 1?


Setup:

  • Task 2 dependent on Task1

 Challenge:

  •  To Fire Task 2 even on Task 1 Failure

Solution:

  • Do "not" Fail Task 1, Handle exception and set parameters in Task 1 ( as suggested by @Retired_mod ) 
  • In Task 2, Utilize Task 1 parameter value to decide next steps

Hope this helps. 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group