Job cluster support in jobs/runs/submit API
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ01-25-2023 01:22 AM
We are using jobs/runs/submit API of databricks to create and trigger a one-time run with new_cluster and existing_cluster configuration.
We would like to check if there is provision to pass "job_clusters" in this API to reuse the same cluster across multiple tasks of job?
We are not able to find this in Databricks API Doc.
- Labels:
-
API
-
DatabricksJobs
-
Job
-
Job_clusters
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ01-25-2023 03:07 AM
@Hanish Bansalโ
Reading Jobs API updates, they've added support for shared job cluster.
To configure shared job clusters, include a JobCluster array in the JobSettings object. You can specify a maximum of 100 clusters per job. The following is an example of an API 2.1 response for a job configured with two shared clusters
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ01-26-2023 04:57 AM
I tried with a sample but got error "Shared job cluster feature is not supported in runs/submit API."
I found the same in documentation "https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/jobs-api-updates?source=recommendations#--runs-submit" that runs submit request does not support shared job clusters.
Not sure if there is any specific reason to not supporting shared job clusters in this API.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ01-28-2023 06:37 AM
@Hanish Bansalโ Shared job cluster for jobs/runs/submit API is not supported at the moment.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ08-06-2024 01:01 AM
Hi,
Any update on this issue? Is a shared job cluster supported for the jobs/runs/submit API or planned to be supported in the near future?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ09-03-2024 01:15 PM
Hi,
Any update on the above mentioned issue? Unable to submit a one time new job run (api/2.0 or 21/jobs/runs/submit) with shared job cluster or one new cluster has to be used for all TASKs in the job