Can anyone tell me how to submit a job using serverless compute using restapi

tonyd — Fri, 27 Sep 2024 08:29:15 GMT

#serverless#createjob

Re: Can anyone tell me how to submit a job using serverless compute using restapi

filipniziol — Fri, 27 Sep 2024 13:40:08 GMT

Hi @tonyd ,
If you want to use serverless, first of all you need to enable it:

When enabled, when sending the request to the REST API simply do not specify the cluster:

import requests import json # Your Databricks domain DATABRICKS_DOMAIN = '<>' # Personal Access Token for authentication TOKEN = '<>' # The JSON string representing your job configuration job_json_string = """{ "name": "my_serverless_job", "email_notifications": { "no_alert_for_skipped_runs": false }, "webhook_notifications": {}, "timeout_seconds": 0, "max_concurrent_runs": 1, "tasks": [ { "task_key": "df_regular", "run_if": "ALL_SUCCESS", "notebook_task": { "notebook_path": "/Workspace/Users/my_user/df_regular", "base_parameters": { "storage_account": "my_storage_account" }, "source": "WORKSPACE" }, "timeout_seconds": 0, "email_notifications": {}, "notification_settings": { "no_alert_for_skipped_runs": false, "no_alert_for_canceled_runs": false, "alert_on_last_attempt": false }, "webhook_notifications": {} } ], "run_as": { "user_name": "my_user" } }""" # Convert the JSON string to a dictionary job_config = json.loads(job_json_string) # The API endpoint for creating a job api_url = f'{DATABRICKS_DOMAIN}/api/2.1/jobs/create' # Headers for the request headers = { 'Authorization': f'Bearer {TOKEN}', 'Content-Type': 'application/json' } # Make the POST request to create the job response = requests.post(api_url, headers=headers, json=job_config) # Check if the request was successful if response.status_code == 200: # Job created successfully print("Job created successfully.") # Print the job ID job_id = response.json()['job_id'] print(f"Job ID: {job_id}") else: # There was an error print("Failed to create job.") print(f"Status Code: {response.status_code}") print(f"Response: {response.text}")

The job has been created:

The job compute is serverless:

Hope it helps

topic Can anyone tell me how to submit a job using serverless compute using restapi in Data Engineering

Can anyone tell me how to submit a job using serverless compute using restapi

Re: Can anyone tell me how to submit a job using serverless compute using restapi