topic Re: "Run now with different parameters" - different parameters not recognized by jobs invo in Data Engineering

"Run now with different parameters" - different parameters not recognized by jobs involving multiple tasks

swzzzsw — Mon, 24 Jan 2022 19:17:24 GMT

I'm running a databricks job involving multiple tasks and would like to run the job with different set of task parameters. I can achieve that by edit each task and and change the parameter values. However, it gets very manual when I have a lot of tasks. I'm think of using "run now with different parameters" option and pass in a different job JSON file. However, it looks like no matter what pass into the JSON file at the job level, my tasks still run with the old parameter values.

Does "run now with different parameters" option support changing parameter values for several tasks at once?

Re: "Run now with different parameters" - different parameters not recognized by jobs involving multiple tasks

Anonymous — Tue, 25 Jan 2022 14:51:33 GMT

Hello, @Shirly Wang! My name is Piper, and I'm a moderator for the Databricks community. Welcome to the community and thank you for your question! It's nice to meet you. Let's give the members of the community a chance to respond before we return to your question.

Thanks for your patience.

Re: "Run now with different parameters" - different parameters not recognized by jobs involving multiple tasks

ally_r — Thu, 25 Aug 2022 14:36:39 GMT

This is an issue for me too. I am using a multi-stage job calling the same notebook with different values for the parameters. This feels like a bug to me.

Re: "Run now with different parameters" - different parameters not recognized by jobs involving multiple tasks

erens — Wed, 14 Dec 2022 12:32:48 GMT

Hello,

I am also facing with the same issue. The problem is described below:

I have a multi-task job. This job consists of multiple "spark_python_task" kind tasks that execute a python script in a spark cluster. This pipeline is created within a CI/CD process and all tasks have their default parameters.
When I want to run this job via Jobs UI by clicking "Run now with different parameters" button, I can see that I can specify only one single parameter set for "spark_python_task". I would like to be able to pass arguments to tasks separately. And when I pass parameters, I am observing that these parameters are passed to all tasks.
Even though it is stated that parameters are merged with the default parameters, I am observing that my parameters are overridden. (reference)

Apart from Jobs UI, I also suffer from the same problem when I want to trigger the job using API (reference).

In short, how can I pass different parameters to tasks and get those parameters merged instead of being overridden in a multi-task environment?

Re: "Run now with different parameters" - different parameters not recognized by jobs invo

VijayNakkonda — Mon, 03 Mar 2025 10:13:02 GMT

Dear Team, For now, I found a solution. Disconnect the bundle source on Databricks, edit the parameters that you want to run. After execution, redeploy your code again from repository.

Re: "Run now with different parameters" - different parameters not recognized by jobs invo

Dali1 — Fri, 06 Mar 2026 08:44:02 GMT

Hello Anyone found a better solution for this ?