topic Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes? in Data Engineering

Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

elizeu_reis — Tue, 11 Aug 2026 21:02:00 GMT

We've been digging into serverless interactive compute costs in notebooks, because our internal cost monitoring (cross-referencing system.billing.usage with system.query.history) showed a large share of the spend coming from idle time rather than actual query execution. To understand why, we wanted to inspect the actual cluster configuration behind a serverless notebook session — but the notebook UI doesn't expose a cluster ID anywhere for serverless, so there's no obvious way to query it via API or SDK.

How we found the serverless cluster ID (not exposed in the UI):

The workaround: open your browser's DevTools → Network tab, attach the notebook to Serverless, and run any cell. This triggers a request to the internal notebook/{notebook_id}/command endpoint. In that request's headers, look at x-databricks-attribution-tags — it's a URL-encoded JSON blob, and one of its fields (clusterId) contains the serverless cluster ID generated for that session. With that ID, you can query the cluster programmatically via the SDK:

import os from databricks.sdk import WorkspaceClient from databricks.sdk.service import compute w = WorkspaceClient() cluster_id = "your_serverless_cluster_id" cluster = w.clusters.get(cluster_id) print(cluster.autotermination_minutes)

The actual finding:

Querying the cluster this way, I see that autotermination_minutes defaults to 60 minutes — the same value historically used for classic all-purpose clusters, which take several minutes to spin up (which is why it makes sense not to tear down a classic cluster too quickly).

But serverless compute has the opposite value proposition: near-instant startup, with no provisioning cost for the user to wait through. That should allow for a much more aggressive idle timeout — without the "wait for the cluster to come back up" trade-off that justifies 60 minutes on classic clusters.

In practice, this generates real waste: we cross-referenced system.billing.usage with system.query.history to measure real usage vs. idle time in developer sessions, and in one specific case we found 99.7% of billed time was idle — a user who ran just over 1 minute of actual commands ended up billed for a 6-hour session, because the notebook was left open without ever hitting "Terminate."

When I query interactive serverless sessions for a user directly against the billing table, I can clearly see several sessions lasting exactly 60 minutes.

%sql -- 'Query generated by genie code' WITH prices AS ( SELECT sku_name, usage_unit, price_start_time, COALESCE(price_end_time, date_add(current_date, 1)) AS price_end_time_eff, pricing.default AS unit_price FROM system.billing.list_prices WHERE currency_code = 'USD' ) SELECT u.usage_start_time, u.usage_end_time, ROUND(timestampdiff(SECOND, u.usage_start_time, u.usage_end_time) / 60.0, 2) AS duration_minutes, u.sku_name, ROUND(u.usage_quantity, 4) AS dbu, ROUND(u.usage_quantity * p.unit_price, 4) AS cost_usd, u.usage_metadata.notebook_id AS notebook_id, u.usage_metadata.notebook_path AS notebook_path FROM system.billing.usage u LEFT JOIN prices p ON u.sku_name = p.sku_name AND u.usage_unit = p.usage_unit AND u.usage_end_time BETWEEN p.price_start_time AND p.price_end_time_eff WHERE u.billing_origin_product = 'INTERACTIVE' AND u.identity_metadata.run_as = 'USER_EMAIL' AND u.usage_date = '2026-08-10' AND u.workspace_id IN ('WORKSPACE_ID_1', 'WORKSPACE_ID_2') ORDER BY u.usage_start_time

So, why is the default auto-termination for serverless interactive notebook compute 60 minutes?

Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

elizeu_reis — Wed, 12 Aug 2026 01:52:20 GMT

Confirmed with an actual serverless session left idle until it hit the timeout on its own:

This INACTIVITY termination only happened because I closed the notebook and reopened it more than an hour later. When I left the notebook open (tab still active, no cells running) for over an hour, the serverless cluster did not auto-terminate after the 60-minute idle window. So the idle timeout appears to only kick in reliably when the notebook itself is closed/disconnected, not simply when there's no command execution while the notebook stays open.

Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

Niyojit — Wed, 12 Aug 2026 07:00:00 GMT

Hi @elizeu_reis ,

You can change the Serverless Cluster timeout by going to Settings → Compute.

I’ve attached a screenshot for reference. I hope this helps!

Thanks,

Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

RobertMitchel — Wed, 12 Aug 2026 07:58:36 GMT

The distinction between serverless startup time and classic cluster provisioning is a good point. If serverless can start almost immediately, a 60-minute idle period can definitely result in unnecessary usage for notebooks that are left open.

I’d be interested to know whether the 60-minute value is mainly a product-design choice for interactive workflows or whether there are technical reasons for keeping the default that high. The observation that an open notebook may remain active even when no cells are running is particularly important from a cost-management perspective.

For teams concerned about unexpected usage, it seems worthwhile to monitor system.billing.usage alongside query history and establish internal guidance for disconnecting idle notebook sessions. A shorter configurable timeout would also give administrators more control over this type of workload.

After spending time investigating usage and billing, something lightweight like rollerballer.io can be a quick browser break.

Re: Why is the default auto-termination for serverless interactive notebook compute 60 minutes?

Armanillo_RoK — Wed, 12 Aug 2026 09:30:44 GMT

That's an interesting find @elizeu_reis , let's see what DBX folks come up back with on that.

Here are my two cents re I see that autotermination_minutes defaults to 60 minutes
-> if one happens to be creating a personal compute (not serverless) in Databricks, the initial setting of termination of the cluster set to 4320 mins and (which equals to 3 days...what a lovely Easter egg :D!) So one needs to keep in mind to change it to something around 20 mins instead right away and once done with their work switch the cluster off best.