Is there any better way to manage workflow dependencies using databricks workflows

[128941]

Looking for solution on how to manage job dependencies using databricks workflows.

[Mounika_Tarigop]

Yes, Databricks Workflows provides several ways to manage workflow dependencies:

1. Job Dependencies: You can set up job dependencies in Databricks Workflows. This means that one job will only start after another job has successfully completed. This is useful for managing dependencies between different data processing tasks.

2. Task Dependencies: Within a single job, you can set up task dependencies using the Databricks notebook workflows feature. This allows you to call one notebook from another, creating a dependency between the two notebooks.

[sebasv]

Hi Mounika,
Could you provide links to the documentation for both kinds of dependencies?

[Simen_Holmestad]

+1 on this.

What if we want one job to run when three other jobs are finished? (which is possible for DAGs in Airflow by using datasets)

[jacovangelder]

Have you tried Asset Bundles? You can set both job and task dependencies in here. 

[128941]

Trying to find a way to manage dependencies between the multiple databricks workflows.

Example: wf1-->wf2--wf3 and wk4.

[128941]

We have not tried Asset Bundles.