topic Databricks Cluster Start and Stop in Data Engineering

Databricks Cluster Start and Stop

mailsathesh — Sat, 24 Aug 2024 12:23:47 GMT

I want to send out an email if the cluster failed to start, i used to start the cluster using Databricks cli and then terminate them. In some cases ,my cluster is not starting at all and there are some errors.

My use case is to send an email using databricks cli, start the cluster and terminate it if it started. incase if it fails to start, then send an email to a particular support group email id.

Above is my use case. Can you help me with this or is there any better way to do it? Please suggest me if any

Re: Databricks Cluster Start and Stop

szymon_dybczak — Sat, 24 Aug 2024 13:32:36 GMT

Hi @mailsathesh ,

You can write a script that will use databricks cli to start cluster. You can you --timeout flag to set maximum amount of time to reach running state. If this amoun is exceeded or if there is any error you can then send an email with notification.
Databricsk Cli doesn't have email sending feature, but you can write script in bash/powershel/other scripting language to do that.

PS: You can also use databricks REST API

Re: Databricks Cluster Start and Stop

mailsathesh — Sat, 24 Aug 2024 14:32:10 GMT

Thank you @szymon_dybczak . let me try this and post it back in this channel