https://community.databricks.com/t5/data-engineering/can-we-have-a-depend-on-for-jobs-to-run-on-two-different-dabs/m-p/116264#M45282 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/160888">@vivi007</a>&nbsp;</P><P>Yes, you can create dependencies between jobs in different DABs (Databricks Asset Bundles), but this requires a different approach than task dependencies within a single DAB.</P><P>Since DABs are designed to be independently deployable units, direct dependencies between them aren't built into the DAB structure itself. However, you can implement cross-DAB job dependencies using a few methods:</P><OL><LI><P><STRONG>Databricks Workflows API</STRONG>: You can create a programmatic solution using the Workflows API to check the completion status of a job in one DAB before triggering a job in another DAB.</P></LI><LI><P><STRONG>External orchestration tools</STRONG>: Tools like Apache Airflow can orchestrate workflows across multiple DABs by monitoring job completion and triggering dependent jobs.</P></LI><LI><P><STRONG>Event-based triggers</STRONG>: Set up the first DAB's job to publish an event (e.g., to a webhook or message queue) upon completion, and configure the second DAB's job to start when it receives this event.</P></LI><LI><P><STRONG>Delta Live Tables (DLT) Pipeline Dependencies</STRONG>: If you're using DLT pipelines, you can create dependencies between pipelines that exist in different DABs.</P></LI><LI><P><STRONG>Scheduled dependencies</STRONG>: Configure the dependent job with a schedule that starts after the expected completion time of the prerequisite job, possibly with additional checks.</P></LI></OL><P>Remember that when implementing cross-DAB dependencies, you'll need to handle error scenarios and retries appropriately, as well as ensure proper access controls are in place between the different asset bundles.</P> Wed, 23 Apr 2025 04:14:17 GMT lingareddy_Alva 2025-04-23T04:14:17Z https://community.databricks.com/t5/data-engineering/can-we-have-a-depend-on-for-jobs-to-run-on-two-different-dabs/m-p/116230#M45274 <P>If there are two different DABs. Can we have a dependency for one job from one DAB to run after a job run from another DAB?&nbsp; Similar to how tasks can depend on each other to run one after the other in the same DAB. Can we have the same for two different DABs?</P> Tue, 22 Apr 2025 17:59:01 GMT https://community.databricks.com/t5/data-engineering/can-we-have-a-depend-on-for-jobs-to-run-on-two-different-dabs/m-p/116230#M45274 vivi007 2025-04-22T17:59:01Z https://community.databricks.com/t5/data-engineering/can-we-have-a-depend-on-for-jobs-to-run-on-two-different-dabs/m-p/116264#M45282 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/160888">@vivi007</a>&nbsp;</P><P>Yes, you can create dependencies between jobs in different DABs (Databricks Asset Bundles), but this requires a different approach than task dependencies within a single DAB.</P><P>Since DABs are designed to be independently deployable units, direct dependencies between them aren't built into the DAB structure itself. However, you can implement cross-DAB job dependencies using a few methods:</P><OL><LI><P><STRONG>Databricks Workflows API</STRONG>: You can create a programmatic solution using the Workflows API to check the completion status of a job in one DAB before triggering a job in another DAB.</P></LI><LI><P><STRONG>External orchestration tools</STRONG>: Tools like Apache Airflow can orchestrate workflows across multiple DABs by monitoring job completion and triggering dependent jobs.</P></LI><LI><P><STRONG>Event-based triggers</STRONG>: Set up the first DAB's job to publish an event (e.g., to a webhook or message queue) upon completion, and configure the second DAB's job to start when it receives this event.</P></LI><LI><P><STRONG>Delta Live Tables (DLT) Pipeline Dependencies</STRONG>: If you're using DLT pipelines, you can create dependencies between pipelines that exist in different DABs.</P></LI><LI><P><STRONG>Scheduled dependencies</STRONG>: Configure the dependent job with a schedule that starts after the expected completion time of the prerequisite job, possibly with additional checks.</P></LI></OL><P>Remember that when implementing cross-DAB dependencies, you'll need to handle error scenarios and retries appropriately, as well as ensure proper access controls are in place between the different asset bundles.</P> Wed, 23 Apr 2025 04:14:17 GMT https://community.databricks.com/t5/data-engineering/can-we-have-a-depend-on-for-jobs-to-run-on-two-different-dabs/m-p/116264#M45282 lingareddy_Alva 2025-04-23T04:14:17Z