How to pass Dynamic parameters like job.run_id to a pipeline_task in Databricks workflow job?

[NancyX]

Is it possible to pass dynamic parameters, such as job.run_id to a pipeline_task within a Databricks Workflow job?

[dataminion01]

yes. it's in the Parameters section

[NancyX]

Does this option available for Task type -Pipeline? I am able to see this for example notebook task but not for pipeline task

[Ashwin_DSA]

Hi @NancyX,

Apologies for the delayed follow-up. Closing the loop here with the current state. Yes, pipeline tasks do support parameters, and you can use dynamic value references such as {{job.run_id}} when configuring a pipeline task’s Parameters field. The relevant docs are Pipeline task for jobs, Access parameter values from a task, and What is a dynamic value reference?. The docs also note that pipeline task parameters are in Beta, so if you do not see the Parameters section for a Pipeline task in the UI, that is usually because the feature is gated by preview availability in the workspace.

If you want to pass the workflow run ID to the pipeline, set a pipeline task parameter like run_id = {{job.run_id}}. Inside the pipeline, those values can then be referenced as pipeline parameters; for SQL pipelines, the documented approach is named parameters, while for Python pipeline code, the docs recommend continuing to use the pipeline configuration field for runtime values.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

[mazeem-arbisoft]

Hi @NancyX ,

In addition to @Ashwin_DSA 's explanation, if you need to do the same from Declarative Automation Bundles (DABs formerly) workflow YAML markup, here are a couple examples from one of my jobs.

Run ID:

- name: job_run_id
      default: "{{job.run_id}}"

Referring your own custom job params in task params for example, here is an example of spark task referring to the custom params defined under the workflow job:

tasks:
  - task_key: my_task
    spark_python_task:
      python_file: ../src/my_python_file.py
      parameters:
        - "--is-historical-run"
        - "{{job.parameters.is_historical_run}}"