Hello everyone,
I recently encountered an issue with one of our production all-purpose clusters where the auto-termination feature did not work as expected.
The cluster was configured with "Terminate after 20 minutes of inactivity", but it never terminated. Unfortunately, I was on vacation during this period, and when I returned after five days, I discovered that the cluster had been running continuously for approximately 149 hours.
To investigate, I checked the following:
- There were no active Spark jobs during the period when I expected the cluster to terminate.
- Our weekend workloads are configured to run on separate job compute clusters, so this all-purpose cluster should have remained idle over the weekend.
- I also confirmed with my team that no one had executed any long-running VACUUM or OPTIMIZE commands (or any other operations) that might have kept the cluster active.
This is the first time in nearly three years that we've seen auto-termination fail on this cluster, so I'm trying to understand what might have prevented it from shutting down.
Has anyone experienced something similar? Is there a way to determine what kept the cluster alive or to analyze why auto-termination did not trigger? Are there any logs, metrics, or diagnostic tools that could help identify the root cause?
I've attached:
The cluster configuration.
A screenshot from the Spark UI showing the long-running application.
Any suggestions or guidance on how to troubleshoot this would be greatly appreciated. Thank you!
