topic Re: Cannot create job through Jobs API in Data Engineering

Cannot create job through Jobs API

keenan_jones7 — Thu, 12 Aug 2021 14:50:37 GMT

import requests import json

instance_id = 'abcd.azuredatabricks.net'

api_version = '/api/2.0' api_command = '/jobs/create' url = f"https://{instance_id}{api_version}{api_command}"

headers = {'Authorization': 'Bearer myToken'}

params = { "settings": { "name": "Test API", "new_cluster": { "cluster_name": "", "spark_version": "8.2.x-scala2.12", "spark_conf": { "spark.databricks.delta.preview.enabled": "true", "spark.sql.shuffle.partitions": "1024" }, "node_type_id": "Standard_D16s_v3", "spark_env_vars": { "PYSPARK_PYTHON": "/databricks/python3/bin/python3" }, "enable_elastic_disk": True, "azure_attributes": { "first_on_demand": 1, "availability": "ON_DEMAND_AZURE", "spot_bid_max_price": -1.0 }, "num_workers": 8 }, "email_notifications": { "no_alert_for_skipped_runs": False }, "timeout_seconds": 0, "notebook_task": { "notebook_path": "/Repos/abcd@repo.com/notebook", "base_parameters": { "start_date": "", "end_date": "", "client": "", "client_output_sa": "", "initial_telomere_load": "", "run_validation": "", "run_telomere": "" } }, "max_concurrent_runs": 1, "format": "SINGLE_TASK" }, "creator_user_name": "abcd@repo.com" }

response = requests.post(url = url, params = params, headers = headers)

print(json.dumps(json.loads(response.text), indent = 2))

I am executing the above code (with real instance id, bearer token, and paths) and keep getting the following error:

{
  "error_code": "INVALID_PARAMETER_VALUE",
  "message": "Job settings must be specified."
}

Re: Cannot create job through Jobs API

BilalAslamDbrx — Mon, 06 Sep 2021 09:44:19 GMT

@keenan_jones7 what's happening is that you are not sending the request body correctly. Please take a look at this documentation. Do you see how there is no top-level settings key? You also don't need to specify the format key or creator_user_name.

Re: Cannot create job through Jobs API

rAlex — Wed, 19 Apr 2023 19:49:03 GMT

@keenan_jones7 I had the same problem today. It looks like you've copied and pasted the JSON that Databricks displays in the GUI when you select View JSON from the dropdown menu when viewing a job.

In order to use that JSON in a request to the Jobs API, you want to use just the JSON that is the value of the settings key. In your case, that would be:

{ "name": "Test API", "new_cluster": { "cluster_name": "", "spark_version": "8.2.x-scala2.12", "spark_conf": { "spark.databricks.delta.preview.enabled": "true", "spark.sql.shuffle.partitions": "1024" }, "node_type_id": "Standard_D16s_v3", "spark_env_vars": { "PYSPARK_PYTHON": "/databricks/python3/bin/python3" }, "enable_elastic_disk": True, "azure_attributes": { "first_on_demand": 1, "availability": "ON_DEMAND_AZURE", "spot_bid_max_price": -1.0 }, "num_workers": 8 }, "email_notifications": { "no_alert_for_skipped_runs": False }, "timeout_seconds": 0, "notebook_task": { "notebook_path": "/Repos/abcd@repo.com/notebook", "base_parameters": { "start_date": "", "end_date": "", "client": "", "client_output_sa": "", "initial_telomere_load": "", "run_validation": "", "run_telomere": "" } }, "max_concurrent_runs": 1, "format": "SINGLE_TASK" }