Auto-Termination Did Not Trigger on Production Cluster Despite 20-Minute Inactivity

[van45678]

Hello everyone,

I recently encountered an issue with one of our production all-purpose clusters where the auto-termination feature did not work as expected.

The cluster was configured with "Terminate after 20 minutes of inactivity", but it never terminated. Unfortunately, I was on vacation during this period, and when I returned after five days, I discovered that the cluster had been running continuously for approximately 149 hours.

To investigate, I checked the following:

1. There were no active Spark jobs during the period when I expected the cluster to terminate.
2. Our weekend workloads are configured to run on separate job compute clusters, so this all-purpose cluster should have remained idle over the weekend.
3. I also confirmed with my team that no one had executed any long-running VACUUM or OPTIMIZE commands (or any other operations) that might have kept the cluster active.

This is the first time in nearly three years that we've seen auto-termination fail on this cluster, so I'm trying to understand what might have prevented it from shutting down.

Has anyone experienced something similar? Is there a way to determine what kept the cluster alive or to analyze why auto-termination did not trigger? Are there any logs, metrics, or diagnostic tools that could help identify the root cause?

I've attached:

The cluster configuration.
A screenshot from the Spark UI showing the long-running application.

Any suggestions or guidance on how to troubleshoot this would be greatly appreciated. Thank you!

[balajij8]

The Spark UI details you shared points directly to the cause - Databricks Shell application. Auto termination only triggers when a cluster is completely idle with zero active sessions or connections. Databricks Shell application represents an active connection to the cluster that acts as a continuous heartbeat, resetting the 20 minute inactivity timer even if no actual Spark jobs are executing. You can stop the cluster immediately to stop incurring costs if feasible.

Its most likely caused by an open notebook browser tab. If someone on your team or you left a notebook attached to this cluster open in a browser before vacation, that browser tab will send periodic keep alive pings to the cluster, preventing auto-termination.

Other causes may include a persistent API connection from a databricks-connect client or an SDK.

Your Spark UI indicates there is a second Databricks Shell application that has already been running for 4.9 hours. You need to terminate this to allow the cluster to spin down

• Navigate to the cluster's detail page and click the Notebooks tab.

• Detach any idle notebooks currently connected to the cluster.

To identify what kept the cluster alive and identify who or what initiated the connection, you can follow below

• Cluster Event Log - In the cluster UI, navigate to the Event Log tab and look at the events right around the time termination should have originally occurred. Look for "Starting," "Restarting," or "Edited" events.

• Review Driver Logs - Download the driver logs (log4j output) from the cluster page. Check for keywords like auto-termination, inactivity, connection or session to find the exact heartbeat timestamps.

• Audit Notebook Attachments - Look at the history of which notebooks were attached to this cluster during the 149-hour window. It will usually point you directly to the user who left their tab open.

• Check the SQL Tab - While you checked the Jobs tab, also check the Spark UI's SQL and Streaming tabs to ensure a BI tool or silently failing streaming query wasn't maintaining a JDBC/ODBC connection.

You can strictly Segregate Workloads by using Job compute clusters for automated tasks (Job clusters terminate instantly upon completion) and lock down access to all purpose cluster so it is only used for explicitly approved interactive debugging. Establish a strict protocol for detaching notebooks when wrapping up work for the day before logging off for weekends or PTO.

[van45678]

Thanks @balajij8 for the detailed explanation. I investigated a few of the suggestions and have some follow-up questions.

I have already checked the Cluster Event Log. Since the cluster never terminated during the entire 149-hour period, there were no additional Starting, Restarting, or Edited events after it was initially started. In other words, there doesn't appear to be anything in the event log that explains why auto-termination never triggered.

I also reviewed the driver logs (log4j output) and searched for terms such as auto-termination, inactivity, connection, and session , but I couldn't find anything that indicated what was keeping the cluster alive.

I have a couple of questions:

1. How can I audit notebook attachments for a cluster?

   • You mentioned checking the history of notebooks attached to the cluster to identify who may have left a notebook connected.

   • What is the recommended way to do this?

   • Is this information available through system tables, audit logs, the clusters API, or another mechanism?

   • We are not using Unity Catalog, so I am wondering whether this functionality depends on Unity Catalog or if it is available independently.

2. Could you explain in more detail how an open notebook browser tab prevents auto-termination?

   • My understanding was that if a user simply attaches a notebook to an all-purpose cluster and then logs out or closes their session, the cluster should still auto-terminate after the configured inactivity period, provided no Spark jobs are running.

   • Does merely having a notebook attached keep the Databricks Shell session alive?

   • Does the browser periodically send heartbeat requests to the cluster even when no code is being executed?

   • Under what circumstances can a databricks shell application remain active for days without any user intentionally running code?

I am trying to better understand whether this behaviour is expected by design or whether there are edge cases where an interactive session can unintentionally prevent auto-termination for an extended period.

[balajij8]

Hi @van45678 Please move to unity catalog and use new run time always.

You can check whether an external connection (Power BI / Tableau) or background thread maintained an active handle on the driver.

1. Active JDBC/ODBC Sessions or Execution Contexts

BI tools (Power BI, Tableau, Looker), application connection pools or custom scripts using the Databricks SQL Connector frequently hold persistent execution contexts. Even if no queries are currently executing, an open JDBC/ODBC session or client heartbeat sent to the driver prevents the cluster's auto-termination timer from starting.

2. Active Structured Streaming Queries

A readStream process (such as Auto Loader) running in the background will keep the cluster active indefinitely while waiting for incoming files. These micro-batch streams will often appear as active streams under the Structured Streaming tab in the Spark UI.

3. Web Terminal, SSH or Driver-Level Processes

If a workspace user left an active Databricks Web Terminal session open, an SSH connection attached or spawned a background process (such as a custom Python daemon or subprocess) directly on the driver node, the control plane flags the compute node as active regardless of Spark job state.

4. Legacy DStreams or DBR Inactivity Reporting

If this cluster is on an older Databricks Runtime (DBR) version, keep in mind that older Spark runtimes had known limitations with compute activity reporting. Additionally, legacy DStream workloads are generally known not to report activity correctly to the Databricks control plane, which can prevent auto-termination from evaluating inactivity properly.

To pinpoint what kept the cluster alive during the 149 hours, you can check Driver log4j Logs - Download the historical Spark driver logs (driver log4j / stdout) from the cluster page. Search the logs for InactivityTracker or AutoTermination. The driver log will explicitly output the heartbeat evaluation timestamps and log why the inactivity threshold was reset (e.g., active execution contexts > 0).

[binlogreader]

@van45678 -- @balajij8 's diagnosis matches what your Spark UI shows, so I will just add three things from the operational side, one for the postmortem and two for making sure the next occurrence costs you hours instead of five days.

(1) Driver logs and event logs age out, but the billing system table keeps per-cluster consumption. You can reconstruct exactly what the 149 hours cost:

```sql
SELECT usage_date, sku_name, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_metadata.cluster_id = '<cluster_id>'
GROUP BY ALL
ORDER BY usage_date;
```

Join `system.billing.list_prices` if you want dollars instead of DBUs. We lean on these tables whenever we need measured cost rather than estimates, and for an incident writeup a dollar figure lands better than an hour count.

(2) A cluster that fails to terminate is invisible to job monitoring, because nothing is failing. The signal that does exist is consumption, an all-purpose cluster burning DBUs at 2am on a Saturday. A scheduled query or SQL alert on `system.billing.usage`, grouping all-purpose SKUs by cluster and day with a threshold just above your normal interactive usage, would have flagged this on day one.

(3) You can pin `autotermination_minutes` with a fixed or range rule so nobody quietly raises it. One thing worth noting, the cluster policies API accepts attribute keys it does not support without any validation error. The policy update succeeds, the JSON looks right, and the rule silently does nothing. So after adding any policy rule, create a throwaway cluster under that policy and confirm the setting is actually constrained before you trust it