topic Re: I do not have any Spark jobs running, but my cluster is not getting auto-terminated. in Data Engineering

I do not have any Spark jobs running, but my cluster is not getting auto-terminated.

brickster_2018 — Thu, 24 Jun 2021 06:32:56 GMT

The cluster is Idle and there are no Spark jobs running on the Spark UI. Still I see my cluster is active and not getting terminated.

Re: I do not have any Spark jobs running, but my cluster is not getting auto-terminated.

brickster_2018 — Thu, 24 Jun 2021 06:45:13 GMT

Databricks cluster is treated as active if there are any spark or non-Spark operations running on the cluster. Even though there are no Spark jobs running on the cluster, it's possible to have some driver-specific application code running marking the cluster as non-idle. Driver-specific application code may be something like connecting to a REST API Server, polling a database, etc.

One other use case, where the cluster will be marked non-idle is if the cluster has an active JDBC/ODBC connection that submits queries on a frequent interval.

Re: I do not have any Spark jobs running, but my cluster is not getting auto-terminated.

jaimuhuri110325 — Tue, 04 Nov 2025 03:42:02 GMT

Will I be charged for maintaining an idle cluster. If no jobs are running but the cluster is not yet terminated, will I still be charged for the time until it is terminated?