cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Not able create Job via Jobs api in databricks

Pritam
New Contributor II

I am not able to create jobs via jobs API in databricks.

Error=INVALID_PARAMETER_VALUE: Job settings must be specified.

I simply copied the JSON file and saved it. Loaded the same JSON file and tried to create the job via API but the got the above error.

Another thing I tried was, copying the JSON details and assigning it to a variable in python databricks and trying to create the job, but got the same error.

Settings are there in the JSON, but some how the JSON details we are passing to create job post command are not getting in the right format.

Is there a way where I can simply load the JSON details in the databricks notebook and create the job again? I went through their documentation, but there there is no example like how they are inputting the json file in the post command.

4 REPLIES 4

Ajay-Pandey
Esteemed Contributor III

Use below format in your body section -

{ 
 
  "name": "Nightly model training", 
 
  "new_cluster": { 
 
    "spark_version": "7.3.x-scala2.12", 
 
    "node_type_id": "Standard_D3_v2", 
 
    "num_workers": 10 
 
  }, 
 
  "libraries": [ 
 
    { 
 
      "jar": "dbfs:/my-jar.jar" 
 
    }, 
 
    { 
 
      "maven": { 
 
        "coordinates": "org.jsoup:jsoup:1.7.2" 
 
      } 
 
    } 
 
  ], 
 
  "timeout_seconds": 3600, 
 
  "max_retries": 1, 
 
  "schedule": { 
 
    "quartz_cron_expression": "0 15 22 * * ?", 
 
    "timezone_id": "America/Los_Angeles" 
 
  }, 
 
  "spark_jar_task": { 
 
    "main_class_name": "com.databricks.ComputeModels" 
 
  } 
 
} 

and for header use below one-

image.png

Ajay Kumar Pandey

Pritam
New Contributor II

My Header and authorization is working fine and i don't want to hard code the body of the json. I am already loading the json and also copying the json in databricks and assigning to a variable.

By the way your above code works but i don't want to hard code it. I already have some jobs and want to duplicate it in other databrick instance

rAlex
New Contributor III

@Pritam Aryaโ€‹  I had the same problem today. In order to use the JSON that you can get from the GUI in an existing job, in a request to the Jobs API, you want to use just the JSON that is the value of the settings key.

takomyr
New Contributor II

databricks jobs get <JOB_ID> -o json | jq '.settings' > /tmp/job.json
edit /tmp/job.json # job name, etc.
databricks jobs create --json @/tmp/job.json

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group