Hi @Brian Labrom, To pass parameters to your Databricks job via the API, you can use the --conf option when launching a job.
You'll need to modify the notebook_task configuration in your job to pass these parameters as arguments to the notebook.
First, you can access the parameters in your notebook using dbutils.widgets.get().
Here's an example:
full_refresh = dbutils.widgets.get("full_refresh") == "true"
config1 = dbutils.widgets.get("config1")
config2 = dbutils.widgets.get("config2")
Now, when you submit the job through the API, pass the parameters in the
notebook_task section like this:
{
"name": "My Job",
"new_cluster": {
"spark_version": "x.x.x-scala2.x",
"node_type_id": "node_type",
"num_workers": 1
},
"notebook_task": {
"notebook_path": "/path/to/your/notebook",
"base_parameters": {
"full_refresh": "true",
"config1": "config1_value",
"config2": "config2_value"
}
}
}
Replace /path/to/your/notebook with the path to your laptop, and modify the spark_version, node_type_id, and num_workers according to your requirements.
If you're using Python or a different language for your API call, make sure to adjust the code accordingly. For example, in Python, you could use the requests library to submit the job like this:
import json
import requests
api_key = "your_databricks_token"
api_url = "https://your_databricks_instance/api/2.0/jobs/runs/submit"
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}
job_config = {
"name": "My Job",
"new_cluster": {
"spark_version": "x.x.x-scala2.x",
"node_type_id": "node_type",
"num_workers": 1
},
"notebook_task": {
"notebook_path": "/path/to/your/notebook",
"base_parameters": {
"full_refresh": "true",
"config1": "config1_value",
"config2": "config2_value"
}
}
}
response = requests.post(api_url, headers=headers, data=json.dumps(job_config))
Remember to replace your_databricks_token, your_databricks_instance, and other placeholders with your actual values.
