I have a job with some tasks. Some of the tasks are pipeline_task's some are notebook_task's.
When I run the job with "Run now with different parameters" and enter a new key-value, I see that the key-value is available in the notebook_task's with dbutils.widgets.get, but is NOT available in my pipelines with neither spark.conf.get nor dbutils.widgets.get.
For example, if I "run now with different parameters" and pass in the key "fav_color" with the value "blue", then dbutils.widgets.get("fav_color") and spark.conf.get("fav_color") do NOT exist in my pipeline_task. In my notebook_task, I AM able to get the value from dbutils.widgets.get("fav_color").
My question is: How do I get these new parameters in my pipeline_task that were added via "run now with different parameters"?