topic Re: Job cluster support in jobs/runs/submit API in Data Engineering

Job cluster support in jobs/runs/submit API

hanish — Wed, 25 Jan 2023 09:22:35 GMT

We are using jobs/runs/submit API of databricks to create and trigger a one-time run with new_cluster and existing_cluster configuration.

We would like to check if there is provision to pass "job_clusters" in this API to reuse the same cluster across multiple tasks of job?

We are not able to find this in Databricks API Doc.

Re: Job cluster support in jobs/runs/submit API

daniel_sahal — Wed, 25 Jan 2023 11:07:38 GMT

@Hanish Bansal

Reading Jobs API updates, they've added support for shared job cluster.

https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/jobs-api-updates?source=recommendations#api-changes

To configure shared job clusters, include a JobCluster array in the JobSettings object. You can specify a maximum of 100 clusters per job. The following is an example of an API 2.1 response for a job configured with two shared clusters

Re: Job cluster support in jobs/runs/submit API

hanish — Thu, 26 Jan 2023 12:57:36 GMT

I tried with a sample but got error "Shared job cluster feature is not supported in runs/submit API."

I found the same in documentation "https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/jobs-api-updates?source=recommendations#--runs-submit" that runs submit request does not support shared job clusters.

Not sure if there is any specific reason to not supporting shared job clusters in this API.

Re: Job cluster support in jobs/runs/submit API

Anonymous — Sat, 28 Jan 2023 14:37:19 GMT

@Hanish Bansal Shared job cluster for jobs/runs/submit API is not supported at the moment.

Re: Job cluster support in jobs/runs/submit API

giladba — Tue, 06 Aug 2024 08:01:03 GMT

Hi,
Any update on this issue? Is a shared job cluster supported for the jobs/runs/submit API or planned to be supported in the near future?

Re: Job cluster support in jobs/runs/submit API

Nagrjuna — Tue, 03 Sep 2024 20:15:37 GMT

Hi,

Any update on the above mentioned issue? Unable to submit a one time new job run (api/2.0 or 21/jobs/runs/submit) with shared job cluster or one new cluster has to be used for all TASKs in the job