We've been digging into serverless interactive compute costs in notebooks, because our internal cost monitoring (cross-referencing system.billing.usage with system.query.history) showed a large share of the spend coming from idle time rather than actual query execution. To understand why, we wanted to inspect the actual cluster configuration behind a serverless notebook session — but the notebook UI doesn't expose a cluster ID anywhere for serverless, so there's no obvious way to query it via API or SDK.
How we found the serverless cluster ID (not exposed in the UI):
The workaround: open your browser's DevTools → Network tab, attach the notebook to Serverless, and run any cell. This triggers a request to the internal notebook/{notebook_id}/command endpoint. In that request's headers, look at x-databricks-attribution-tags — it's a URL-encoded JSON blob, and one of its fields (clusterId) contains the serverless cluster ID generated for that session. With that ID, you can query the cluster programmatically via the SDK:
import os
from databricks.sdk import WorkspaceClient
from databricks.sdk.service import compute
w = WorkspaceClient()
cluster_id = "your_serverless_cluster_id"
cluster = w.clusters.get(cluster_id)
print(cluster.autotermination_minutes)The actual finding:
Querying the cluster this way, I see that autotermination_minutes defaults to 60 minutes — the same value historically used for classic all-purpose clusters, which take several minutes to spin up (which is why it makes sense not to tear down a classic cluster too quickly).
But serverless compute has the opposite value proposition: near-instant startup, with no provisioning cost for the user to wait through. That should allow for a much more aggressive idle timeout — without the "wait for the cluster to come back up" trade-off that justifies 60 minutes on classic clusters.
In practice, this generates real waste: we cross-referenced system.billing.usage with system.query.history to measure real usage vs. idle time in developer sessions, and in one specific case we found 99.7% of billed time was idle — a user who ran just over 1 minute of actual commands ended up billed for a 6-hour session, because the notebook was left open without ever hitting "Terminate."
When I query interactive serverless sessions for a user directly against the billing table, I can clearly see several sessions lasting exactly 60 minutes.
%sql
-- 'Query generated by genie code'
WITH prices AS (
SELECT
sku_name,
usage_unit,
price_start_time,
COALESCE(price_end_time, date_add(current_date, 1)) AS price_end_time_eff,
pricing.default AS unit_price
FROM system.billing.list_prices
WHERE currency_code = 'USD'
)
SELECT
u.usage_start_time,
u.usage_end_time,
ROUND(timestampdiff(SECOND, u.usage_start_time, u.usage_end_time) / 60.0, 2) AS duration_minutes,
u.sku_name,
ROUND(u.usage_quantity, 4) AS dbu,
ROUND(u.usage_quantity * p.unit_price, 4) AS cost_usd,
u.usage_metadata.notebook_id AS notebook_id,
u.usage_metadata.notebook_path AS notebook_path
FROM system.billing.usage u
LEFT JOIN prices p
ON u.sku_name = p.sku_name
AND u.usage_unit = p.usage_unit
AND u.usage_end_time BETWEEN p.price_start_time AND p.price_end_time_eff
WHERE
u.billing_origin_product = 'INTERACTIVE'
AND u.identity_metadata.run_as = 'USER_EMAIL'
AND u.usage_date = '2026-08-10'
AND u.workspace_id IN ('WORKSPACE_ID_1', 'WORKSPACE_ID_2')
ORDER BY u.usage_start_timeSo, why is the default auto-termination for serverless interactive notebook compute 60 minutes?
Confirmed with an actual serverless session left idle until it hit the timeout on its own:
This INACTIVITY termination only happened because I closed the notebook and reopened it more than an hour later. When I left the notebook open (tab still active, no cells running) for over an hour, the serverless cluster did not auto-terminate after the 60-minute idle window. So the idle timeout appears to only kick in reliably when the notebook itself is closed/disconnected, not simply when there's no command execution while the notebook stays open.
