topic start another workflow waiting the completion of a job-run of the same workflow in Data Engineering

start another workflow waiting the completion of a job-run of the same workflow

jeremy98 — Tue, 26 Nov 2024 15:02:55 GMT

Hello community,
I'm using DABs I want to know if It is possible to configure the yaml file a logic that allows me to run a workflow if the previous job run is finished of the same workflow. Is it possible to do it? Do I need to create a task that checks this job execution?

Thanks to everyone,

Re: start another workflow waiting the completion of a job-run of the same workflow

Alberto_Umana — Tue, 26 Nov 2024 16:55:07 GMT

Hello @jeremy98,

Yes, it is possible to configure a YAML file to run a workflow only if the previous job run of the same workflow has finished. You can achieve this by defining dependencies between tasks within the workflow.

You can specify task dependencies using the depends_on field in your YAML configuration. This ensures that a task will only run after the specified dependent tasks have completed successfully

Example code:

bundle:

name: example-bundle

resources:

jobs:

example-job:

name: example-job

tasks:

- task_key: task1

notebook_task:

notebook_path: /path/to/notebook1

- task_key: task2

depends_on:

- task_key: task1

notebook_task:

notebook_path: /path/to/notebook2

Please see: https://github.com/databricks/bundle-examples/blob/6558a732f4f21e2de7440aa0aadfd9ed8f845395/default_python/resources/default_python_job.yml

Re: start another workflow waiting the completion of a job-run of the same workflow

jeremy98 — Tue, 26 Nov 2024 17:07:03 GMT

Thanks @Alberto_Umana for this solution, but I mean I need to execute only one notebook task in the job, so the depends_on isn't possible to set. Do u mean that I need to create a first task that is a check_monitoring_job that checks if any other job is still running and later when it is finished execute the last one (my interested ingestion task)?