Is there any better way to manage workflow dependencies using databricks workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-12-2024 12:29 PM
Looking for solution on how to manage job dependencies using databricks workflows.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-27-2024 03:49 PM
Yes, Databricks Workflows provides several ways to manage workflow dependencies:
-
Job Dependencies: You can set up job dependencies in Databricks Workflows. This means that one job will only start after another job has successfully completed. This is useful for managing dependencies between different data processing tasks.
-
Task Dependencies: Within a single job, you can set up task dependencies using the Databricks notebook workflows feature. This allows you to call one notebook from another, creating a dependency between the two notebooks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-25-2024 02:10 AM
Hi Mounika,
Could you provide links to the documentation for both kinds of dependencies?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-27-2024 05:23 AM
+1 on this.
What if we want one job to run when three other jobs are finished? (which is possible for DAGs in Airflow by using datasets)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-28-2024 12:53 AM
Have you tried Asset Bundles? You can set both job and task dependencies in here.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-19-2024 11:30 AM
Trying to find a way to manage dependencies between the multiple databricks workflows.
Example: wf1-->wf2--wf3 and wk4.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-19-2024 11:31 AM
We have not tried Asset Bundles.

