cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Run now with different parameters doesn't pass parameter to pipeline tasks

Jaku6
New Contributor

I have a job with some tasks. Some of the tasks are pipeline_task's some are notebook_task's.

When I run the job with "Run now with different parameters" and enter a new key-value, I see that the key-value is available in the notebook_task's with dbutils.widgets.get, but is NOT available in my pipelines with neither spark.conf.get nor dbutils.widgets.get.

For example, if I "run now with different parameters" and pass in the key "fav_color" with the value "blue", then dbutils.widgets.get("fav_color") and spark.conf.get("fav_color") do NOT exist in my pipeline_task. In my notebook_task, I AM able to get the value from dbutils.widgets.get("fav_color").

My question is: How do I get these new parameters in my pipeline_task that were added via "run now with different parameters"?

1 REPLY 1

Walter_C
Honored Contributor

As per docs it seems that pipeline task type is currently not supported to pass parameters: https://docs.databricks.com/en/jobs/create-run-jobs.html#pass-parameters-to-a-databricks-job-task

You could create a notebook task that runs before your pipeline task. This notebook task would take the job parameters and write them to a location accessible by your pipeline task, such as DBFS (Databricks File System) or a database. Then, your pipeline task could read these parameters from that location.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group