topic Re: Is there any better way to manage workflow dependencies using databricks workflows in Data Engineering

Is there any better way to manage workflow dependencies using databricks workflows

128941 — Wed, 12 Jun 2024 19:29:39 GMT

Looking for solution on how to manage job dependencies using databricks workflows.

Mounika_Tarigop — Thu, 27 Jun 2024 22:49:38 GMT

Yes, Databricks Workflows provides several ways to manage workflow dependencies:

Job Dependencies: You can set up job dependencies in Databricks Workflows. This means that one job will only start after another job has successfully completed. This is useful for managing dependencies between different data processing tasks.
Task Dependencies: Within a single job, you can set up task dependencies using the Databricks notebook workflows feature. This allows you to call one notebook from another, creating a dependency between the two notebooks.

jacovangelder — Fri, 28 Jun 2024 07:53:38 GMT

Have you tried Asset Bundles? You can set both job and task dependencies in here.

128941 — Fri, 19 Jul 2024 18:30:36 GMT

Trying to find a way to manage dependencies between the multiple databricks workflows.

Example: wf1-->wf2--wf3 and wk4.

128941 — Fri, 19 Jul 2024 18:31:28 GMT

We have not tried Asset Bundles.

sebasv — Fri, 25 Oct 2024 09:10:56 GMT

Hi Mounika,
Could you provide links to the documentation for both kinds of dependencies?

Simen_Holmestad — Wed, 27 Nov 2024 13:23:06 GMT

+1 on this.

What if we want one job to run when three other jobs are finished? (which is possible for DAGs in Airflow by using datasets)