cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Cannot create job through Jobs API

keenan_jones7
New Contributor II

import requests import json

instance_id = 'abcd.azuredatabricks.net'

api_version = '/api/2.0' api_command = '/jobs/create' url = f"https://{instance_id}{api_version}{api_command}"

headers = {'Authorization': 'Bearer myToken'}

params = { "settings": { "name": "Test API", "new_cluster": { "cluster_name": "", "spark_version": "8.2.x-scala2.12", "spark_conf": { "spark.databricks.delta.preview.enabled": "true", "spark.sql.shuffle.partitions": "1024" }, "node_type_id": "Standard_D16s_v3", "spark_env_vars": { "PYSPARK_PYTHON": "/databricks/python3/bin/python3" }, "enable_elastic_disk": True, "azure_attributes": { "first_on_demand": 1, "availability": "ON_DEMAND_AZURE", "spot_bid_max_price": -1.0 }, "num_workers": 8 }, "email_notifications": { "no_alert_for_skipped_runs": False }, "timeout_seconds": 0, "notebook_task": { "notebook_path": "/Repos/abcd@repo.com/notebook", "base_parameters": { "start_date": "", "end_date": "", "client": "", "client_output_sa": "", "initial_telomere_load": "", "run_validation": "", "run_telomere": "" } }, "max_concurrent_runs": 1, "format": "SINGLE_TASK" }, "creator_user_name": "abcd@repo.com" }

response = requests.post(url = url, params = params, headers = headers)

print(json.dumps(json.loads(response.text), indent = 2))

I am executing the above code (with real instance id, bearer token, and paths) and keep getting the following error:
{
  "error_code": "INVALID_PARAMETER_VALUE",
  "message": "Job settings must be specified."
} 

3 REPLIES 3

BilalAslamDbrx
Honored Contributor III

@keenan_jones7​  what's happening is that you are not sending the request body correctly. Please take a look at this documentation. Do you see how there is no top-level settings key? You also don't need to specify the format key or creator_user_name.

Kaniz_Fatma
Community Manager
Community Manager

Hi @keenan_jones7​ , Just a friendly follow-up. Do you still need help, or @Bilal Aslam​'s response helps you to find the solution? Please let us know.

rAlex
New Contributor III

@keenan_jones7​ I had the same problem today. It looks like you've copied and pasted the JSON that Databricks displays in the GUI when you select View JSON from the dropdown menu when viewing a job.

In order to use that JSON in a request to the Jobs API, you want to use just the JSON that is the value of the settings key. In your case, that would be:

{ "name": "Test API", "new_cluster": { "cluster_name": "", "spark_version": "8.2.x-scala2.12", "spark_conf": { "spark.databricks.delta.preview.enabled": "true", "spark.sql.shuffle.partitions": "1024" }, "node_type_id": "Standard_D16s_v3", "spark_env_vars": { "PYSPARK_PYTHON": "/databricks/python3/bin/python3" }, "enable_elastic_disk": True, "azure_attributes": { "first_on_demand": 1, "availability": "ON_DEMAND_AZURE", "spot_bid_max_price": -1.0 }, "num_workers": 8 }, "email_notifications": { "no_alert_for_skipped_runs": False }, "timeout_seconds": 0, "notebook_task": { "notebook_path": "/Repos/abcd@repo.com/notebook", "base_parameters": { "start_date": "", "end_date": "", "client": "", "client_output_sa": "", "initial_telomere_load": "", "run_validation": "", "run_telomere": "" } }, "max_concurrent_runs": 1, "format": "SINGLE_TASK" }

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group