Louis_Frolio
Databricks Employee
Databricks Employee

Hi @elizeu_reis , great detective work here. Cross-referencing system.billing.usage with system.query.history is exactly the right way to separate real usage from idle burn. I did some digging, and I can confirm most of what you observed. There are two different timeouts getting tangled up in this thread, so let's pull them apart.

The control @Niyojit  pointed to (Settings > Compute > Serverless interactive execution timeout) caps how long a single statement can run before it gets cancelled. The default is 2.5 hours (9,000 seconds), admins can change it at the workspace level, and you can override it per notebook with spark.databricks.execution.timeout. It's a good guardrail against runaway queries, but it has nothing to do with idle session termination. That's why your 300 second setting had no effect on the 60 minute INACTIVITY termination. Docs here: https://docs.databricks.com/aws/en/compute/serverless/notebooks

Now, the direct answer to your question: today there is no supported user or admin control for the idle timeout on serverless notebook sessions. The platform manages the session lifecycle. The docs acknowledge that idle termination happens, but they don't publish the value or expose a knob. Your measurement of 60 minutes with termination_reason.code = INACTIVITY matches what others have seen. My colleague @Ashwin_DSA dug into the same question in a related thread, confirmed there's no setting to tune, and used the same billing table approach to verify charges: https://community.databricks.com/t5/data-engineering/serverless-notebook-idle-timeout-is-it-configur...

As for why the default sits at 60 minutes, I honestly don't know, and I haven't found a published rationale. @RobertMitchel asked whether it's a product design choice or a technical constraint, and I won't guess on the record. What I can say is that on serverless the trade-off isn't startup time, it's session state. Terminating the session tears down your REPL, and every variable, temp view, and cached DataFrame goes with it. A generous idle window has historically been the protection against that.

Which brings me to the piece that changes the math: Automated session restoration for serverless notebooks. Turn it on under Settings > Developer > Experimental features. Databricks snapshots the notebook's Python and Spark state before idle termination, and when you come back a Reconnect banner restores your variables, functions, DataFrames, and temp views. Take a gander at https://www.databricks.com/blog/seamlessly-resume-sessions-serverless-notebooks and https://docs.databricks.com/aws/en/notebooks/notebook-compute. Once losing state stops hurting, terminating early (manually or otherwise) gets a lot easier to live with.

On your second observation, that an open tab kept the session alive past 60 minutes: that matches what others have reported. Termination appears to key off the notebook being disconnected, not off command execution alone. So any internal guidance you write should say "close the tab or hit Terminate," not "just stop running cells."

One caveat on the 99.7% figure. system.billing.usage is the source of truth for what you paid, and system.query.history is good activity context, but the two tables have different purposes and granularity. The comparison is a solid estimate of time with no recorded query, not an exact idle-charge ledger. Still plenty to make your case. References: https://docs.databricks.com/aws/en/admin/system-tables/billing and https://docs.databricks.com/aws/en/admin/usage/system-tables

Here's what I'd do today:

  1. Publish internal guidance to hit Terminate (or close the notebook) when done, paired with the session restoration toggle so nobody fears losing their work.
  2. Keep the execution timeout configured as a guardrail against runaway commands, knowing it won't touch the idle issue.
  3. Keep the billing monitor you built, broken out by notebook, user, SKU, and time window. Your query is a good template for others. (@RobertMitchel made the same point about monitoring.)
  4. Raise the ask through your Databricks account team or a support ticket so the product team can track it. Include your cloud and region, workspace IDs, notebook ID and path, timestamps, SKU, and the observed termination reason. You're not the first to ask for a configurable idle value, and the honest framing is a product and documentation gap: is 60 minutes intentional, and if so, can admins get a supported way to reduce it?

Also, +1 to @Armanillo_RoK's related gotcha: classic Personal Compute defaults auto-termination to 4320 minutes (3 days), so change that one right away. Just keep in mind classic auto-termination is a different compute-management model and doesn't carry over to serverless sessions.

One small caution: the x-databricks-attribution-tags header trick pokes at an internal endpoint. Smart way to investigate, but it's an implementation detail, not a supported API contract, and it could change without notice. I wouldn't build tooling on it.

Net takeaway: your diagnosis is right. The idle timeout on serverless notebooks is fixed at the platform level today, the execution timeout is a separate control, and until a knob exists your best tools are terminate discipline, session restoration, and the billing table.

Regards, Louis