topic Re: How to restart Databricks Cluster at specific time? in Data Engineering

How to restart Databricks Cluster at specific time?

Trushna — Thu, 22 Sep 2022 16:43:59 GMT

Command available for restart but not at specific time.

databricks clusters restart --cluster-id <>

Re: How to restart Databricks Cluster at specific time?

Prabakar — Thu, 22 Sep 2022 23:09:52 GMT

Hi @Trushna Khatri , I dont see an option to restart the cluster at a specific time. However, you can achieve this by scheduling a cron job in your terminal. By this, you can set the time to restart your cluster from cli.

The below cron job will restart the cluster daily at 00:00 hrs (mid-night)

0 0 * * * databricks clusters restart --cluster-id 1234-567890-batch123

Re: How to restart Databricks Cluster at specific time?

karthik_p — Mon, 26 Sep 2022 20:07:59 GMT

@Trushna Khatri adding some more information to prabakar. can you please let me know what is actual need of starting cluster during specific time. usually if you criteria is to use for jobs go with job cluster. here cluster start when ever your job start and stops when ever you job gets completed

Re: How to restart Databricks Cluster at specific time?

Trushna — Mon, 26 Sep 2022 21:25:24 GMT

Actually few pipelines are failed and there is some GC issue. right now manually restart the cluster so need to automate the restart cluster at 8 PM daily. Main thing is as per data coming continuously we cant stop and then start the cluster as we have 30 mins window time