cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

passing job parameters through the terminal to a job

om_bk_00
Visitor

I am having troubles overriding the job parameters that are deployed in my local workspace.

e.g I have a job that fills tables with data,

the parameters given to it are random and I would like to override them when I run through my terminal

databricks bundle run <job_name> <NEW_PARAMETERS>

havent been able to do it yet. But i am able to do it in python using DatabricksRunNowOperator through the json

2 REPLIES 2

VZLA
Databricks Employee
Databricks Employee

Hello, @om_bk_00 thank you for your question.

Could you clarify a few details to help me assist you better? Are you encountering any specific errors when trying to override job parameters via the terminal? Also, are you using the databricks bundle run command exclusively, or have you tried the databricks jobs run-now command? If possible, share the command you're running and the parameters you're trying to pass.

To override parameters via the terminal, the databricks jobs run-now command is typically used:

databricks jobs run-now --job-id <job_id> --notebook-params '{"param1": "value1", "param2": "value2"}'

Replace <job_id> with your job's ID and update the JSON object with the new parameters. Let me know if you’re still having trouble or need help with a specific error!

I was trying to do something like

 
databricks bundle run xxx-xxx --var="table_name=$source_table"
 
then I get an error saying that the parameter table_name is not defined, so I guess I misunderstood how overriding parameters work and how "var" is defined.
 
I will try your method and see if it works 🙂 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group