Auto-Termination Did Not Trigger on Production Cluster Despite 20-Minute Inactivity

[van45678]

Hello everyone,

I recently encountered an issue with one of our production all-purpose clusters where the auto-termination feature did not work as expected.

The cluster was configured with "Terminate after 20 minutes of inactivity", but it never terminated. Unfortunately, I was on vacation during this period, and when I returned after five days, I discovered that the cluster had been running continuously for approximately 149 hours.

To investigate, I checked the following:

1. There were no active Spark jobs during the period when I expected the cluster to terminate.
2. Our weekend workloads are configured to run on separate job compute clusters, so this all-purpose cluster should have remained idle over the weekend.
3. I also confirmed with my team that no one had executed any long-running VACUUM or OPTIMIZE commands (or any other operations) that might have kept the cluster active.

This is the first time in nearly three years that we've seen auto-termination fail on this cluster, so I'm trying to understand what might have prevented it from shutting down.

Has anyone experienced something similar? Is there a way to determine what kept the cluster alive or to analyze why auto-termination did not trigger? Are there any logs, metrics, or diagnostic tools that could help identify the root cause?

I've attached:

The cluster configuration.
A screenshot from the Spark UI showing the long-running application.

Any suggestions or guidance on how to troubleshoot this would be greatly appreciated. Thank you!

[balajij8]

The Spark UI details you shared points directly to the cause - Databricks Shell application. Auto termination only triggers when a cluster is completely idle with zero active sessions or connections. Databricks Shell application represents an active connection to the cluster that acts as a continuous heartbeat, resetting the 20 minute inactivity timer even if no actual Spark jobs are executing. You can stop the cluster immediately to stop incurring costs if feasible.

Its most likely caused by an open notebook browser tab. If someone on your team or you left a notebook attached to this cluster open in a browser before vacation, that browser tab will send periodic keep alive pings to the cluster, preventing auto-termination.

Other causes may include a persistent API connection from a databricks-connect client or an SDK.

Your Spark UI indicates there is a second Databricks Shell application that has already been running for 4.9 hours. You need to terminate this to allow the cluster to spin down

• Navigate to the cluster's detail page and click the Notebooks tab.

• Detach any idle notebooks currently connected to the cluster.

To identify what kept the cluster alive and identify who or what initiated the connection, you can follow below

• Cluster Event Log - In the cluster UI, navigate to the Event Log tab and look at the events right around the time termination should have originally occurred. Look for "Starting," "Restarting," or "Edited" events.

• Review Driver Logs - Download the driver logs (log4j output) from the cluster page. Check for keywords like auto-termination, inactivity, connection or session to find the exact heartbeat timestamps.

• Audit Notebook Attachments - Look at the history of which notebooks were attached to this cluster during the 149-hour window. It will usually point you directly to the user who left their tab open.

• Check the SQL Tab - While you checked the Jobs tab, also check the Spark UI's SQL and Streaming tabs to ensure a BI tool or silently failing streaming query wasn't maintaining a JDBC/ODBC connection.

You can strictly Segregate Workloads by using Job compute clusters for automated tasks (Job clusters terminate instantly upon completion) and lock down access to all purpose cluster so it is only used for explicitly approved interactive debugging. Establish a strict protocol for detaching notebooks when wrapping up work for the day before logging off for weekends or PTO.

[binlogreader]

@van45678 -- @balajij8 's diagnosis matches what your Spark UI shows, so I will just add three things from the operational side, one for the postmortem and two for making sure the next occurrence costs you hours instead of five days.

(1) Driver logs and event logs age out, but the billing system table keeps per-cluster consumption. You can reconstruct exactly what the 149 hours cost:

```sql
SELECT usage_date, sku_name, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_metadata.cluster_id = '<cluster_id>'
GROUP BY ALL
ORDER BY usage_date;
```

Join `system.billing.list_prices` if you want dollars instead of DBUs. We lean on these tables whenever we need measured cost rather than estimates, and for an incident writeup a dollar figure lands better than an hour count.

(2) A cluster that fails to terminate is invisible to job monitoring, because nothing is failing. The signal that does exist is consumption, an all-purpose cluster burning DBUs at 2am on a Saturday. A scheduled query or SQL alert on `system.billing.usage`, grouping all-purpose SKUs by cluster and day with a threshold just above your normal interactive usage, would have flagged this on day one.

(3) You can pin `autotermination_minutes` with a fixed or range rule so nobody quietly raises it. One thing worth noting, the cluster policies API accepts attribute keys it does not support without any validation error. The policy update succeeds, the JSON looks right, and the rule silently does nothing. So after adding any policy rule, create a throwaway cluster under that policy and confirm the setting is actually constrained before you trust it