Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

elizeu_reis — Tue, 11 Aug 2026 21:02:00 GMT

We've been digging into serverless interactive compute costs in notebooks, because our internal cost monitoring (cross-referencing system.billing.usage with system.query.history) showed a large share of the spend coming from idle time rather than actual query execution. To understand why, we wanted to inspect the actual cluster configuration behind a serverless notebook session — but the notebook UI doesn't expose a cluster ID anywhere for serverless, so there's no obvious way to query it via API or SDK.

How we found the serverless cluster ID (not exposed in the UI):

The workaround: open your browser's DevTools → Network tab, attach the notebook to Serverless, and run any cell. This triggers a request to the internal notebook/{notebook_id}/command endpoint. In that request's headers, look at x-databricks-attribution-tags — it's a URL-encoded JSON blob, and one of its fields (clusterId) contains the serverless cluster ID generated for that session. With that ID, you can query the cluster programmatically via the SDK:

import os from databricks.sdk import WorkspaceClient from databricks.sdk.service import compute w = WorkspaceClient() cluster_id = "your_serverless_cluster_id" cluster = w.clusters.get(cluster_id) print(cluster.autotermination_minutes)

The actual finding:

Querying the cluster this way, I see that autotermination_minutes defaults to 60 minutes — the same value historically used for classic all-purpose clusters, which take several minutes to spin up (which is why it makes sense not to tear down a classic cluster too quickly).

But serverless compute has the opposite value proposition: near-instant startup, with no provisioning cost for the user to wait through. That should allow for a much more aggressive idle timeout — without the "wait for the cluster to come back up" trade-off that justifies 60 minutes on classic clusters.

In practice, this generates real waste: we cross-referenced system.billing.usage with system.query.history to measure real usage vs. idle time in developer sessions, and in one specific case we found 99.7% of billed time was idle — a user who ran just over 1 minute of actual commands ended up billed for a 6-hour session, because the notebook was left open without ever hitting "Terminate."

When I query interactive serverless sessions for a user directly against the billing table, I can clearly see several sessions lasting exactly 60 minutes.

%sql -- 'Query generated by genie code' WITH prices AS ( SELECT sku_name, usage_unit, price_start_time, COALESCE(price_end_time, date_add(current_date, 1)) AS price_end_time_eff, pricing.default AS unit_price FROM system.billing.list_prices WHERE currency_code = 'USD' ) SELECT u.usage_start_time, u.usage_end_time, ROUND(timestampdiff(SECOND, u.usage_start_time, u.usage_end_time) / 60.0, 2) AS duration_minutes, u.sku_name, ROUND(u.usage_quantity, 4) AS dbu, ROUND(u.usage_quantity * p.unit_price, 4) AS cost_usd, u.usage_metadata.notebook_id AS notebook_id, u.usage_metadata.notebook_path AS notebook_path FROM system.billing.usage u LEFT JOIN prices p ON u.sku_name = p.sku_name AND u.usage_unit = p.usage_unit AND u.usage_end_time BETWEEN p.price_start_time AND p.price_end_time_eff WHERE u.billing_origin_product = 'INTERACTIVE' AND u.identity_metadata.run_as = 'USER_EMAIL' AND u.usage_date = '2026-08-10' AND u.workspace_id IN ('WORKSPACE_ID_1', 'WORKSPACE_ID_2') ORDER BY u.usage_start_time

So, why is the default auto-termination for serverless interactive notebook compute 60 minutes?

Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

elizeu_reis — Wed, 12 Aug 2026 01:52:20 GMT

Confirmed with an actual serverless session left idle until it hit the timeout on its own:

This INACTIVITY termination only happened because I closed the notebook and reopened it more than an hour later. When I left the notebook open (tab still active, no cells running) for over an hour, the serverless cluster did not auto-terminate after the 60-minute idle window. So the idle timeout appears to only kick in reliably when the notebook itself is closed/disconnected, not simply when there's no command execution while the notebook stays open.

Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

Niyojit — Wed, 12 Aug 2026 07:00:00 GMT

Hi @elizeu_reis ,

You can change the Serverless Cluster timeout by going to Settings → Compute.

I’ve attached a screenshot for reference. I hope this helps!

Thanks,

topic Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes? in Data Engineering

Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes?