topic Re: passing job parameters through the terminal to a job in Data Engineering

passing job parameters through the terminal to a job

om_bk_00 — Fri, 10 Jan 2025 10:08:25 GMT

I am having troubles overriding the job parameters that are deployed in my local workspace.

e.g I have a job that fills tables with data,

the parameters given to it are random and I would like to override them when I run through my terminal

databricks bundle run <job_name> <NEW_PARAMETERS>

havent been able to do it yet. But i am able to do it in python using DatabricksRunNowOperator through the json

Re: passing job parameters through the terminal to a job

VZLA — Fri, 10 Jan 2025 10:29:27 GMT

Hello, @om_bk_00 thank you for your question.

Could you clarify a few details to help me assist you better? Are you encountering any specific errors when trying to override job parameters via the terminal? Also, are you using the databricks bundle run command exclusively, or have you tried the databricks jobs run-now command? If possible, share the command you're running and the parameters you're trying to pass.

To override parameters via the terminal, the databricks jobs run-now command is typically used:

databricks jobs run-now --job-id <job_id> --notebook-params '{"param1": "value1", "param2": "value2"}'

Replace <job_id> with your job's ID and update the JSON object with the new parameters. Let me know if you’re still having trouble or need help with a specific error!

Re: passing job parameters through the terminal to a job

om_bk_00 — Fri, 10 Jan 2025 10:46:57 GMT

I was trying to do something like

databricks bundle run xxx-xxx --var="table_name=$source_table"

then I get an error saying that the parameter table_name is not defined, so I guess I misunderstood how overriding parameters work and how "var" is defined.

I will try your method and see if it works 🙂

Re: passing job parameters through the terminal to a job

om_bk_00 — Mon, 13 Jan 2025 08:56:39 GMT

Ended up using this and it worked 🙂

databricks bundle run <job_name> --python-params "--param1=$source_table,..."

The only challenge was to figure out how the params needs to be written exactly as they are in the parameterized job; so it took a few trials. Because a list in python looks a bit different than a list in the parameters on databricks (spaces, etc...)

Re: passing job parameters through the terminal to a job

VZLA — Mon, 13 Jan 2025 10:07:31 GMT

Excellent! Thanks for sharing your working version and solution!

Re: passing job parameters through the terminal to a job

EduardoSB — Wed, 16 Jul 2025 07:01:57 GMT

Hi! I just found this post because I'm having troubles trying to pass custom values to some parameters in my jobs. I guess

databricks bundle run <job_name> --python-params "--param1=value1,--param2=value2,..."

should work, shouldn't it? Is any other extra configuration necessary? E.g. does the yaml to deploy the job need to have some specific structure, something like the example I attach below (see at the very end of the yaml, the bold part, please)?

resources:

jobs:

my_job:

name: my_job

...

tasks:

- task_key: main_task

job_cluster_key: job_cluster

python_wheel_task:

package_name: my_package

entry_point: main

libraries:

- whl: ../dist/*.whl

python_params:

param1: ...

param2: ...

...