topic Re: Creating a Databricks Asset Bundle with Sequential Pipelines and Workflow using YAML in Data Engineering

Creating a Databricks Asset Bundle with Sequential Pipelines and Workflow using YAML

smit_tw — Thu, 05 Dec 2024 19:01:55 GMT

Is it possible to create a repository with a Databricks asset bundle that includes the following pipelines?

Test1 (Delta Live Table Pipeline)
Test2 (Delta Live Table Pipeline)
Test3 (Delta Live Table Pipeline)
Workflow Job
Workflow to execute the above pipelines in sequence (4 → 1 → 2 → 3).

Can you create 5 YAML files that accomplish the following:

Define and set up the pipelines.
Configure the workflow to run them in the specified sequence (4 → 1 → 2 → 3), ensuring each pipeline correctly references the ones it depends on?

Re: Creating a Databricks Asset Bundle with Sequential Pipelines and Workflow using YAML

filipniziol — Thu, 05 Dec 2024 19:34:24 GMT

Hi @smit_tw ,

yes, it is possible to do so, here is the sample yml:

resources: jobs: sample_job: name: sample_job tasks: - task_key: JobTask run_job_task: job_id: 1094194179990459 - task_key: DltTask1 depends_on: - task_key: JobTask pipeline_task: pipeline_id: da5fa00c-33b6-4850-8ea3-53f6e8d4b0e9 full_refresh: false - task_key: DltTask2 depends_on: - task_key: DltTask1 pipeline_task: pipeline_id: da5fa00c-33b6-4850-8ea3-53f6e8d4b0e9 full_refresh: false queue: enabled: true

The possible improvement is to use lookup to to reference jobs and pipelines by their name rather than their ids.

Check this discussion:
https://community.databricks.com/t5/data-engineering/getting-job-id-dynamically-to-create-another-job-to-refer-as-job/td-p/88843

Re: Creating a Databricks Asset Bundle with Sequential Pipelines and Workflow using YAML

smit_tw — Fri, 06 Dec 2024 14:03:47 GMT

Hello @filipniziol , Thank you so much that link is what I was looking for.

Re: Creating a Databricks Asset Bundle with Sequential Pipelines and Workflow using YAML

filipniziol — Fri, 06 Dec 2024 16:18:25 GMT

Hi @smit_tw,

Great! If this resolves your question, please consider marking it as the solution. It helps others in the community find answers more easily. 😊