cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Tutorial docs for running a job using serverless?

mrstevegross
New Contributor III

I'm exploring whether serverless (https://docs.databricks.com/en/jobs/run-serverless-jobs.html#create-a-job-using-serverless-compute) could be useful for our use case. I'd like to see an example of using serverless via the API. The docs say "To learn about using the Jobs API to create and run jobs that use serverless compute, see Jobs in the REST API reference". But that's just reference info, not a tutorial. Is there any tutorial-style documentation showing how to start a job using serverless?

2 ACCEPTED SOLUTIONS

Accepted Solutions

mrstevegross
New Contributor III

>this by default will make the job serverless

Aha, very interesting. Do the reference docs (https://docs.databricks.com/api/workspace/jobs_21/create#tasks) state that? If not, can y'all add an explicit mention in the docs?

View solution in original post

Walter_C
Databricks Employee
Databricks Employee

No, as of now it is not possible 

View solution in original post

7 REPLIES 7

Walter_C
Databricks Employee
Databricks Employee

What specifically you need, is it how you define the job to run with serverless? If this is the case, when creating a job if you need to use serverless you just dont specify cluster configuration, this by default will make the job serverless

Walter_C
Databricks Employee
Databricks Employee

To create a serverless job using the API, you no longer need to specify one of new_cluster, existing_cluster_id, or job_cluster_key in each task. Instead, only tasks with task_key and your task to run is required. Here is an example of how you can create a serverless job:

{
 "name": "Serverless Job",
 "tasks": [
  {
   "task_key": "My_task",
   "python_wheel_task": {
    "package_name": "databricks_jaws",
    "entry_point": "run_analysis",
    "named_parameters": {
     "dry_run": "true"
    }
   },
   "environment_key": "my-serverless-compute"
  }
 ],
 "tags": {
  "department": "sales"
 },
 "environments": [
  {
   "environment_key": "default",
   "spec": {
    "client": "1",
    "dependencies": [
     "/Volumes/<catalog>/<schema>/<volume>/<path>.whl",
     "/Workspace/my_project/dist.whl",
     "simplejson",
     "-r /Workspace/my_project/requirements.txt"
    ]
   }
  }
 ]
}

In this example, the job is named "Serverless Job" and it has a single task with the key "My_task". The task is a Python wheel task that runs the "run_analysis" function from the "databricks_jaws" package. The task is run in the "my-serverless-compute" environment. The job also has a tag indicating that it belongs to the "sales" department.

mrstevegross
New Contributor III

>this by default will make the job serverless

Aha, very interesting. Do the reference docs (https://docs.databricks.com/api/workspace/jobs_21/create#tasks) state that? If not, can y'all add an explicit mention in the docs?

Walter_C
Databricks Employee
Databricks Employee

I will share this feedback as this is currently not stated there

mrstevegross
New Contributor III

>I will share this feedback as this is currently not stated there

Awesome, thanks!

One follow-up question: is it possible to provide a Docker image as part of serverless? (It appears to be a feature of a *cluster*, not a *job*, so I think the answer is no; however, I just want to make sure I've understood).

Walter_C
Databricks Employee
Databricks Employee

No, as of now it is not possible 

mrstevegross
New Contributor III

Thanks!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group