The Spark UI details you shared points directly to the cause - Databricks Shell application. Auto termination only triggers when a cluster is completely idle with zero active sessions or connections. Databricks Shell application represents an active connection to the cluster that acts as a continuous heartbeat, resetting the 20 minute inactivity timer even if no actual Spark jobs are executing. You can stop the cluster immediately to stop incurring costs if feasible.

Its most likely caused by an open notebook browser tab. If someone on your team or you left a notebook attached to this cluster open in a browser before vacation, that browser tab will send periodic keep alive pings to the cluster, preventing auto-termination.

Other causes may include a persistent API connection from a databricks-connect client or an SDK.

Your Spark UI indicates there is a second Databricks Shell application that has already been running for 4.9 hours. You need to terminate this to allow the cluster to spin down

• Navigate to the cluster's detail page and click the Notebooks tab.

• Detach any idle notebooks currently connected to the cluster.

To identify what kept the cluster alive and identify who or what initiated the connection, you can follow below

• Cluster Event Log - In the cluster UI, navigate to the Event Log tab and look at the events right around the time termination should have originally occurred. Look for "Starting," "Restarting," or "Edited" events.

• Review Driver Logs - Download the driver logs (log4j output) from the cluster page. Check for keywords like auto-termination, inactivity, connection or session to find the exact heartbeat timestamps.

• Audit Notebook Attachments - Look at the history of which notebooks were attached to this cluster during the 149-hour window. It will usually point you directly to the user who left their tab open.

• Check the SQL Tab - While you checked the Jobs tab, also check the Spark UI's SQL and Streaming tabs to ensure a BI tool or silently failing streaming query wasn't maintaining a JDBC/ODBC connection.

You can strictly Segregate Workloads by using Job compute clusters for automated tasks (Job clusters terminate instantly upon completion) and lock down access to all purpose cluster so it is only used for explicitly approved interactive debugging. Establish a strict protocol for detaching notebooks when wrapping up work for the day before logging off for weekends or PTO.