Job cluster support in jobs/runs/submit API

hanish
New Contributor II

‎01-25-2023 01:22 AM

We are using jobs/runs/submit API of databricks to create and trigger a one-time run with new_cluster and existing_cluster configuration.

We would like to check if there is provision to pass "job_clusters" in this API to reuse the same cluster across multiple tasks of job?

We are not able to find this in Databricks API Doc.

Labels:

daniel_sahal
Databricks MVP

‎01-25-2023 03:07 AM

@Hanish Bansal​ 

Reading Jobs API updates, they've added support for shared job cluster.

https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/jobs-api-updates?source=recommenda...

To configure shared job clusters, include a JobCluster array in the JobSettings object. You can specify a maximum of 100 clusters per job. The following is an example of an API 2.1 response for a job configured with two shared clusters

0 Kudos

hanish
New Contributor II
In response to daniel_sahal

‎01-26-2023 04:57 AM

I tried with a sample but got error "Shared job cluster feature is not supported in runs/submit API."

I found the same in documentation "https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/jobs-api-updates?source=recommendations#--runs-submit" that runs submit request does not support shared job clusters.

Not sure if there is any specific reason to not supporting shared job clusters in this API.

0 Kudos

Anonymous
Not applicable

‎01-28-2023 06:37 AM

@Hanish Bansal​ Shared job cluster for  jobs/runs/submit API is not supported at the moment.

0 Kudos

giladba
New Contributor III

‎08-06-2024 01:01 AM

Hi, 
Any update on this issue? Is a shared job cluster supported for the jobs/runs/submit API or planned to be supported in the near future?

0 Kudos

Nagrjuna
New Contributor II

‎09-03-2024 01:15 PM

Hi, 

Any update on the above mentioned issue? Unable to submit a one time new job run (api/2.0 or 21/jobs/runs/submit) with shared job cluster or one new cluster has to be used for all TASKs in the job

 

0 Kudos