topic Re: Scheduling multiple jobs (workflows) in DABs in Get Started Discussions

Scheduling multiple jobs (workflows) in DABs

Greg_c — Mon, 27 Jan 2025 11:15:43 GMT

Hello, I'm wondering how can I schedule multiple jobs (workflow).

I'd like to do something like this but on a workflow level.

tasks: - task_key: task_1 sql_task: warehouse_id: ${var.warehouse_id} parameters: ... file: path: ../src/task_1.sql - task_key: task_2 depends_on: - task_key: drop_tables sql_task: warehouse_id: ${var.warehouse_id} parameters: ... file: path: ../src/task_2.sql

and what I want is a master workflow like

jobs/workflows: - job/workflow_key: workflow_1 file: path: ../resources/workflow_1.sql - job/workflow_key: workflow_2 depends_on: - workflow_key: workflow_1 file: path: ../resources/workflow_2.sql

I'd really appreciate any resources/examples on this - thank you!

Re: Scheduling multiple jobs (workflows) in DABs

Alberto_Umana — Mon, 27 Jan 2025 12:38:51 GMT

Hi @Greg_c,

You can try with this sctructure:

- In the main databricks.yml

# databricks.yml
bundle:
name: master-bundle

include:
- resources/*.yml

# Other bundle configurations...

In resource directory, create a YAML for each job:

# resources/job1.yml
resources:
jobs:
job1:
name: First Job
tasks:
- task_key: task1
notebook_task:
notebook_path: /path/to/notebook1

# resources/job2.yml
resources:
jobs:
job2:
name: Second Job
tasks:
- task_key: task2
notebook_task:
notebook_path: /path/to/notebook2

Re: Scheduling multiple jobs (workflows) in DABs

Greg_c — Mon, 27 Jan 2025 14:33:12 GMT

How exactly do I configure it here?

# databricks.yml bundle: name: master-bundle include: - resources/*.yml

In what order will this run these jobs? I want to be able to set the dependencies etc. myself. Please provide whole code for this examples