topic Not able create Job via Jobs api in databricks in Data Engineering

Not able create Job via Jobs api in databricks

Pritam — Thu, 22 Dec 2022 09:29:39 GMT

I am not able to create jobs via jobs API in databricks.

Error=INVALID_PARAMETER_VALUE: Job settings must be specified.

I simply copied the JSON file and saved it. Loaded the same JSON file and tried to create the job via API but the got the above error.

Another thing I tried was, copying the JSON details and assigning it to a variable in python databricks and trying to create the job, but got the same error.

Settings are there in the JSON, but some how the JSON details we are passing to create job post command are not getting in the right format.

Is there a way where I can simply load the JSON details in the databricks notebook and create the job again? I went through their documentation, but there there is no example like how they are inputting the json file in the post command.

Re: Not able create Job via Jobs api in databricks

Ajay-Pandey — Thu, 22 Dec 2022 09:48:30 GMT

Use below format in your body section -

{ 
 
  "name": "Nightly model training", 
 
  "new_cluster": { 
 
    "spark_version": "7.3.x-scala2.12", 
 
    "node_type_id": "Standard_D3_v2", 
 
    "num_workers": 10 
 
  }, 
 
  "libraries": [ 
 
    { 
 
      "jar": "dbfs:/my-jar.jar" 
 
    }, 
 
    { 
 
      "maven": { 
 
        "coordinates": "org.jsoup:jsoup:1.7.2" 
 
      } 
 
    } 
 
  ], 
 
  "timeout_seconds": 3600, 
 
  "max_retries": 1, 
 
  "schedule": { 
 
    "quartz_cron_expression": "0 15 22 * * ?", 
 
    "timezone_id": "America/Los_Angeles" 
 
  }, 
 
  "spark_jar_task": { 
 
    "main_class_name": "com.databricks.ComputeModels" 
 
  } 
 
}

and for header use below one-

Re: Not able create Job via Jobs api in databricks

Pritam — Thu, 22 Dec 2022 13:34:34 GMT

My Header and authorization is working fine and i don't want to hard code the body of the json. I am already loading the json and also copying the json in databricks and assigning to a variable.

By the way your above code works but i don't want to hard code it. I already have some jobs and want to duplicate it in other databrick instance

Re: Not able create Job via Jobs api in databricks

rAlex — Wed, 19 Apr 2023 19:52:06 GMT

@Pritam Arya I had the same problem today. In order to use the JSON that you can get from the GUI in an existing job, in a request to the Jobs API, you want to use just the JSON that is the value of the settings key.

Re: Not able create Job via Jobs api in databricks

takomyr — Tue, 13 Aug 2024 15:13:06 GMT

databricks jobs get <JOB_ID> -o json | jq '.settings' > /tmp/job.json
edit /tmp/job.json # job name, etc.
databricks jobs create --json @/tmp/job.json