cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Passing parameters (variables?) in DAGs

Greg_c
New Contributor II

Regarding DAGs and tasks in them - can I pass a parameter/variable in a task?

I have the same structure like here: https://github.com/databricks/bundle-examples/blob/main/default_sql/resources/default_sql_sql_job.ym...

and I want to pass variables to .sql file from tasks. It works when I put that as a parameter in parameters section like:

 

resources:
  jobs:
    some_workflow:
      name: some_workflow

      trigger:
        .....

      parameters:
        - name: catalog
          default: ${var.catalog}
        - name: schema
          default: ${var.schema}
        - name: bundle_target
          default: ${bundle.target}
        - name: table_to_drop
          default: name_of_table_to_drop

      tasks:
        - task_key: drop_table
          sql_task:
            warehouse_id: ${var.warehouse_id}
            file:
              path: ../src/drop_table.sql

 

but if I want to specify that as a parameter/variable within task like this:

 

      tasks:
        - task_key: process_data
          parameters:
            input_table: ${parameters.source_table}
          variables: 
            environment: ${var.env}

 

it doesn't work.
Am I missing something? I've gone through all docs possible but didn't find an answer to that.

1 REPLY 1

filipniziol
Contributor III

Hi @Greg_c ,

In Databricks Asset Bundles you have a possibility to pass parameter to SQL File Task.

Here is end to end example:

1. My SQL File (with :id parameter):

 

filipniziol_0-1737824991732.png

2. The job YAML:

resources:
  jobs:
    run_sql_file_job:
      name: run_sql_file_job
      tasks:
        - task_key: run_sql_file_task
          sql_task:
            parameters:
              id: "1"
            file:
              path: /Workspace/Shared/sql_file_with_parameter.sql
              source: WORKSPACE
            warehouse_id: b277bba4a65e4c55
      queue:
        enabled: true

3. The result:

filipniziol_1-1737825144470.png

4. The difference between your code and the example is you define parameters one level up compared to what is the working yaml:

      tasks:
        - task_key: process_data
          parameters:
            input_table: ${parameters.source_table}

vs.

      tasks:
        - task_key: run_sql_file_task
          sql_task:
            parameters:
              id: "1"

Let me know if that helps.

 

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group