topic Re: What happens to the clusters whose jobs are canceled or terminated due to failures? (Jobs triggered through Job API2.1 using runs/submit) in Data Engineering

What happens to the clusters whose jobs are canceled or terminated due to failures? (Jobs triggered through Job API2.1 using runs/submit)

Junee — Mon, 22 Nov 2021 10:28:07 GMT

I am using Databeicks Job Api 2.1 to trigger and run my jobs. "jobs/runs/submit" this API helps in starting the cluster, as well as create the job and run it. This API works great for normal jobs as it also cleans the cluster once job is finished successfully.

I know through my POC that in case when the jobs are not successful (Failed, Cancelled or Terminated), the cluster created by the API is maintained in the Job Clusters UI.

I want to know that for how long these Job Clusters are maintained by default? Or are they maintained unless until they are cleaned manually?

Thanks for the read and help on this.

Re: What happens to the clusters whose jobs are canceled or terminated due to failures? (Jobs triggered through Job API2.1 using runs/submit)

Hubert-Dudek — Mon, 22 Nov 2021 12:58:49 GMT

This is really interesting question. I bet that better is to use pool for servers for that jobs tasks so we can see in pool how servers behave and also have settings for that. From my experience VM exists for at least few minutes after job failed (when pool have min idle 0). What technically is going there after job failed in that case should rather be answered by someone from inside of databricks @Kaniz Fatma

Re: What happens to the clusters whose jobs are canceled or terminated due to failures? (Jobs triggered through Job API2.1 using runs/submit)

User16871418122 — Wed, 24 Nov 2021 06:46:10 GMT

The job clusters for finished or failed runs are maintained in Job Clusters UI. They are up to 30 recently terminated job clusters are retained in UI and others are terminated. The finished or canceled runs are also cleaned up automatically starting with the oldest terminated cluster first.

One more thing to note, these terminated clusters list maintained in the Datbricks UI is just for config reference or audit of recent runs. These clusters do not link to any cloud resources (VMs, IPs, or Disks). These resources do not incur any actual cost in the cloud.

Re: What happens to the clusters whose jobs are canceled or terminated due to failures? (Jobs triggered through Job API2.1 using runs/submit)

Junee — Thu, 25 Nov 2021 04:10:28 GMT

Hi @Gobinath Viswanathan , Thank you for your detailed answer.

Is this something you know from experience or do you have a source of this information from official documentations?

Re: What happens to the clusters whose jobs are canceled or terminated due to failures? (Jobs triggered through Job API2.1 using runs/submit)

User16871418122 — Thu, 25 Nov 2021 04:14:22 GMT

@Junee, Anytime! 🙂 It is crisply mentioned in the doc too. https://docs.databricks.com/clusters/index.html

Re: What happens to the clusters whose jobs are canceled or terminated due to failures? (Jobs triggered through Job API2.1 using runs/submit)

Junee — Thu, 25 Nov 2021 04:28:04 GMT

Gracias @Gobinath Viswanathan 🙂