Creating a Databricks Asset Bundle with Sequential Pipelines and Workflow using YAML

[smit_tw]

Is it possible to create a repository with a Databricks asset bundle that includes the following pipelines?

1. Test1 (Delta Live Table Pipeline)
2. Test2 (Delta Live Table Pipeline)
3. Test3 (Delta Live Table Pipeline)
4. Workflow Job
5. Workflow to execute the above pipelines in sequence (4 → 1 → 2 → 3).

Can you create 5 YAML files that accomplish the following:

• Define and set up the pipelines.
• Configure the workflow to run them in the specified sequence (4 → 1 → 2 → 3), ensuring each pipeline correctly references the ones it depends on?

[filipniziol · Accepted solution]

Hi @smit_tw ,

yes, it is possible to do so, here is the sample yml:

resources:
  jobs:
    sample_job:
      name: sample_job
      tasks:
        - task_key: JobTask
          run_job_task:
            job_id: 1094194179990459
        - task_key: DltTask1
          depends_on:
            - task_key: JobTask
          pipeline_task:
            pipeline_id: da5fa00c-33b6-4850-8ea3-53f6e8d4b0e9
            full_refresh: false
        - task_key: DltTask2
          depends_on:
            - task_key: DltTask1
          pipeline_task:
            pipeline_id: da5fa00c-33b6-4850-8ea3-53f6e8d4b0e9
            full_refresh: false
      queue:
        enabled: true

The possible improvement is to use lookup to to reference jobs and pipelines by their name rather than their ids.

Check this discussion:
https://community.databricks.com/t5/data-engineering/getting-job-id-dynamically-to-create-another-jo...

[smit_tw]

Hello @filipniziol , Thank you so much that link is what I was looking for. 

[filipniziol]

Hi @smit_tw,

Great! If this resolves your question, please consider marking it as the solution. It helps others in the community find answers more easily. 😊