What happens to the clusters whose jobs are canceled or terminated due to failures? (Jobs triggered through Job API2.1 using runs/submit)

[Junee]

I am using Databeicks Job Api 2.1 to trigger and run my jobs. "jobs/runs/submit" this API helps in starting the cluster, as well as create the job and run it. This API works great for normal jobs as it also cleans the cluster once job is finished successfully.

I know through my POC that in case when the jobs are not successful (Failed, Cancelled or Terminated), the cluster created by the API is maintained in the Job Clusters UI.

I want to know that for how long these Job Clusters are maintained by default? Or are they maintained unless until they are cleaned manually?

Thanks for the read and help on this.

[Hubert-Dudek]

This is really interesting question. I bet that better is to use pool for servers for that jobs tasks so we can see in pool how servers behave and also have settings for that. From my experience VM exists for at least few minutes after job failed (when pool have min idle 0). What technically is going there after job failed in that case should rather be answered by someone from inside of databricks @Kaniz Fatma​ 

[Kaniz]

Hi @Hubert Dudek​ , Let's see if someone from the community comes up with an answer. If not I'll get back. Thanks.

[Kaniz]

Hi @Junaid Khan​ ! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[User16871418122 · Accepted solution]

The job clusters for finished or failed runs are maintained in Job Clusters UI. They are up to 30 recently terminated job clusters are retained in UI and others are terminated. The finished or canceled runs are also cleaned up automatically starting with the oldest terminated cluster first.

One more thing to note, these terminated clusters list maintained in the Datbricks UI is just for config reference or audit of recent runs. These clusters do not link to any cloud resources (VMs, IPs, or Disks). These resources do not incur any actual cost in the cloud.

[Junee]

Hi @Gobinath Viswanathan​ , Thank you for your detailed answer.

Is this something you know from experience or do you have a source of this information from official documentations?

[User16871418122]

@Junee, Anytime! 🙂 It is crisply mentioned in the doc too. https://docs.databricks.com/clusters/index.html

[Junee]

Gracias @Gobinath Viswanathan  🙂