Passing parameters (variables?) in DAGs

Greg_c — Thu, 23 Jan 2025 15:47:12 GMT

Regarding DAGs and tasks in them - can I pass a parameter/variable in a task?

I have the same structure like here: https://github.com/databricks/bundle-examples/blob/main/default_sql/resources/default_sql_sql_job.yml

and I want to pass variables to .sql file from tasks. It works when I put that as a parameter in parameters section like:

resources: jobs: some_workflow: name: some_workflow trigger: ..... parameters: - name: catalog default: ${var.catalog} - name: schema default: ${var.schema} - name: bundle_target default: ${bundle.target} - name: table_to_drop default: name_of_table_to_drop tasks: - task_key: drop_table sql_task: warehouse_id: ${var.warehouse_id} file: path: ../src/drop_table.sql

but if I want to specify that as a parameter/variable within task like this:

tasks: - task_key: process_data parameters: input_table: ${parameters.source_table} variables: environment: ${var.env}

it doesn't work.
Am I missing something? I've gone through all docs possible but didn't find an answer to that.

Re: Passing parameters (variables?) in DAGs

filipniziol — Sat, 25 Jan 2025 17:14:43 GMT

Hi @Greg_c ,

In Databricks Asset Bundles you have a possibility to pass parameter to SQL File Task.

Here is end to end example:

1. My SQL File (with :id parameter):

2. The job YAML:

resources: jobs: run_sql_file_job: name: run_sql_file_job tasks: - task_key: run_sql_file_task sql_task: parameters: id: "1" file: path: /Workspace/Shared/sql_file_with_parameter.sql source: WORKSPACE warehouse_id: b277bba4a65e4c55 queue: enabled: true

3. The result:

4. The difference between your code and the example is you define parameters one level up compared to what is the working yaml:

tasks: - task_key: process_data parameters: input_table: ${parameters.source_table}

vs.

tasks: - task_key: run_sql_file_task sql_task: parameters: id: "1"

Let me know if that helps.

topic Passing parameters (variables?) in DAGs in Data Engineering

Passing parameters (variables?) in DAGs

Re: Passing parameters (variables?) in DAGs