cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

passing job parameters through the terminal to a job

om_bk_00
New Contributor III

I am having troubles overriding the job parameters that are deployed in my local workspace.

e.g I have a job that fills tables with data,

the parameters given to it are random and I would like to override them when I run through my terminal

databricks bundle run <job_name> <NEW_PARAMETERS>

havent been able to do it yet. But i am able to do it in python using DatabricksRunNowOperator through the json

1 ACCEPTED SOLUTION

Accepted Solutions

om_bk_00
New Contributor III

Ended up using this and it worked ๐Ÿ™‚ 

databricks bundle run <job_name> --python-params "--param1=$source_table,..."

The only challenge was to figure out how the params needs to be written exactly as they are in the parameterized job; so it took a few trials. Because a list in python looks a bit different than a list in the parameters on databricks (spaces, etc...)

View solution in original post

4 REPLIES 4

VZLA
Databricks Employee
Databricks Employee

Hello, @om_bk_00 thank you for your question.

Could you clarify a few details to help me assist you better? Are you encountering any specific errors when trying to override job parameters via the terminal? Also, are you using the databricks bundle run command exclusively, or have you tried the databricks jobs run-now command? If possible, share the command you're running and the parameters you're trying to pass.

To override parameters via the terminal, the databricks jobs run-now command is typically used:

databricks jobs run-now --job-id <job_id> --notebook-params '{"param1": "value1", "param2": "value2"}'

Replace <job_id> with your job's ID and update the JSON object with the new parameters. Let me know if youโ€™re still having trouble or need help with a specific error!

om_bk_00
New Contributor III

I was trying to do something like

 
databricks bundle run xxx-xxx --var="table_name=$source_table"
 
then I get an error saying that the parameter table_name is not defined, so I guess I misunderstood how overriding parameters work and how "var" is defined.
 
I will try your method and see if it works ๐Ÿ™‚ 

om_bk_00
New Contributor III

Ended up using this and it worked ๐Ÿ™‚ 

databricks bundle run <job_name> --python-params "--param1=$source_table,..."

The only challenge was to figure out how the params needs to be written exactly as they are in the parameterized job; so it took a few trials. Because a list in python looks a bit different than a list in the parameters on databricks (spaces, etc...)

VZLA
Databricks Employee
Databricks Employee

Excellent! Thanks for sharing your working version and solution!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group