Passing parameters (variables?) in DAGs

[Greg_c]

Regarding DAGs and tasks in them - can I pass a parameter/variable in a task?

I have the same structure like here: https://github.com/databricks/bundle-examples/blob/main/default_sql/resources/default_sql_sql_job.ym...

and I want to pass variables to .sql file from tasks. It works when I put that as a parameter in parameters section like:

resources:
  jobs:
    some_workflow:
      name: some_workflow

      trigger:
        .....

      parameters:
        - name: catalog
          default: ${var.catalog}
        - name: schema
          default: ${var.schema}
        - name: bundle_target
          default: ${bundle.target}
        - name: table_to_drop
          default: name_of_table_to_drop

      tasks:
        - task_key: drop_table
          sql_task:
            warehouse_id: ${var.warehouse_id}
            file:
              path: ../src/drop_table.sql

but if I want to specify that as a parameter/variable within task like this:

      tasks:
        - task_key: process_data
          parameters:
            input_table: ${parameters.source_table}
          variables: 
            environment: ${var.env}

it doesn't work.
Am I missing something? I've gone through all docs possible but didn't find an answer to that.

[filipniziol]

Hi @Greg_c ,

In Databricks Asset Bundles you have a possibility to pass parameter to SQL File Task.

Here is end to end example:

1. My SQL File (with :id parameter):

2. The job YAML:

resources:
  jobs:
    run_sql_file_job:
      name: run_sql_file_job
      tasks:
        - task_key: run_sql_file_task
          sql_task:
            parameters:
              id: "1"
            file:
              path: /Workspace/Shared/sql_file_with_parameter.sql
              source: WORKSPACE
            warehouse_id: b277bba4a65e4c55
      queue:
        enabled: true

3. The result:

4. The difference between your code and the example is you define parameters one level up compared to what is the working yaml:

      tasks:
        - task_key: process_data
          parameters:
            input_table: ${parameters.source_table}

vs.

      tasks:
        - task_key: run_sql_file_task
          sql_task:
            parameters:
              id: "1"

Let me know if that helps.