cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Can anyone tell me how to submit a job using serverless compute using restapi

tonyd
New Contributor II

#serverless#createjob

1 REPLY 1

filipniziol
Contributor III

Hi @tonyd ,
If you want to use serverless, first of all you need to enable it:

filipniziol_0-1727444114314.png

When enabled, when sending the request to the REST API simply do not specify the cluster:

 

import requests
import json

# Your Databricks domain
DATABRICKS_DOMAIN = '<>'
# Personal Access Token for authentication
TOKEN = '<>'

# The JSON string representing your job configuration
job_json_string =  """{
  "name": "my_serverless_job",
  "email_notifications": {
    "no_alert_for_skipped_runs": false
  },
  "webhook_notifications": {},
  "timeout_seconds": 0,
  "max_concurrent_runs": 1,
  "tasks": [
    {
      "task_key": "df_regular",
      "run_if": "ALL_SUCCESS",
      "notebook_task": {
        "notebook_path": "/Workspace/Users/my_user/df_regular",
        "base_parameters": {
          "storage_account": "my_storage_account"
        },
        "source": "WORKSPACE"
      },
      "timeout_seconds": 0,
      "email_notifications": {},
      "notification_settings": {
        "no_alert_for_skipped_runs": false,
        "no_alert_for_canceled_runs": false,
        "alert_on_last_attempt": false
      },
      "webhook_notifications": {}
    }
  ],
  "run_as": {
    "user_name": "my_user"
  }
}"""

# Convert the JSON string to a dictionary
job_config = json.loads(job_json_string)

# The API endpoint for creating a job
api_url = f'{DATABRICKS_DOMAIN}/api/2.1/jobs/create'

# Headers for the request
headers = {
    'Authorization': f'Bearer {TOKEN}',
    'Content-Type': 'application/json'
}

# Make the POST request to create the job
response = requests.post(api_url, headers=headers, json=job_config)

# Check if the request was successful
if response.status_code == 200:
    # Job created successfully
    print("Job created successfully.")
    # Print the job ID
    job_id = response.json()['job_id']
    print(f"Job ID: {job_id}")
else:
    # There was an error
    print("Failed to create job.")
    print(f"Status Code: {response.status_code}")
    print(f"Response: {response.text}")

 

The job has been created:

filipniziol_1-1727444263334.png

The job compute is serverless:

filipniziol_2-1727444338067.png

Hope it helps

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group