mkassa
New Contributor II

‎11-11-2025 09:27 AM

Hello!
We use a Databricks Job to adjust the cluster’s auto-termination setting based on the day and time. During weekdays from 6 AM to 6 PM, we set auto_termination_minutes to None, which keeps the cluster running. Outside those hours, and on weekends, we set it to 15 minutes so the cluster shuts down quickly when idle. This approach gives us fast turnaround during working hours while still saving costs after hours. You can update the termination settings through the Databricks API or the WorkspaceClient. We do this by running two separate workflows: one sets auto_termination_minutes to None, and the other sets it to 15.

Here is the example notebook code:

 
# Databricks notebook source
from databricks.sdk import WorkspaceClient

# COMMAND ----------

dbutils.widgets.text("auto_termination_minutes", "15", label = "Auto Termination Minutes")

# COMMAND ----------

auto_termination_minutes = dbutils.widgets.get("auto_termination_minutes")

# COMMAND ----------

w = WorkspaceClient()

# Example: get cluster info
clusters = w.clusters.list()
for c in clusters:
    try:
      w.clusters.edit(
          cluster_id = c.cluster_id,
          autotermination_minutes=auto_termination_minutes,
          cluster_name=c.cluster_name,
          spark_version=c.spark_version,
          num_workers=c.num_workers, 
          node_type_id=c.node_type_id,
          spark_conf=c.spark_conf,
          custom_tags=c.custom_tags,
          init_scripts=c.init_scripts,
          cluster_source=c.cluster_source,
          enable_elastic_disk=c.enable_elastic_disk,
          data_security_mode=c.data_security_mode,
          single_user_name=c.single_user_name,
          cluster_log_conf=c.cluster_log_conf,
          spark_env_vars=c.spark_env_vars,
          driver_node_type_id=c.driver_node_type_id,
          policy_id=c.policy_id,
          autoscale=c.autoscale
      )
    except Exception as e:
      pass

 

0 Kudos