Databricks Community

hassanashas · Monday

Hi all, hope everyone is doing well.

Two questions I am hoping someone can help with: is there a way to set an idle timeout for serverless notebooks, and is an attached-but-idle serverless notebook billed while it sits there?

Some background on why I am asking. We have a Python ingest notebook on serverless compute that writes its own run log, so we know precisely how long it ran. When the billed time looked higher than the runtime, we got curious and ran a small experiment to see whether we could account for the difference from system.billing.usage. I have written up what we found below in case it is useful to anyone else, though I would really appreciate a sanity check on whether we have read it correctly.

I should say we may well be missing something obvious, in which case I would be glad to be told so. There is also a thread from March asking something similar which did not quite reach a conclusion, so apologies if this is well-trodden ground.

Setup

Azure Databricks, Premium, US East, serverless compute for notebooks.

A Python notebook that pulls a dataset from a REST API and writes it to a Delta table. It writes its own run log with start and finish timestamps into a separate log table. Screenshot of the log table data is given below:

After the run completed I left the notebook attached rather than detaching it.

What billing recorded

Querying system.billing.usage for that notebook produces two distinct classes of record. Hour-aligned records, and shorter windows that track actual execution:

Total 2.136 DBUs for a 32-minute job.

What we think might be going on

Our reading is that the hour-aligned records are a flat session charge rather than measured compute, though we would be glad to be corrected on this. Three things pointed us that way:

0.750000000000000000 appears exactly, on two different notebooks, for the same full hour. Everything else in the data carries fifteen decimal places.
Nothing in this workspace's serverless history exceeds 0.75 for a full hour. In fairness the workspace is new, so that is hours of history rather than months, and we would not want to read too much into it on its own.
If we treat 0.75 DBU/hr as the rate, every partial record divides out to an attached duration under 60 minutes, which is roughly what we would expect if it were prorated within the hour.

That last point seems to line up with the run log. 0.322279 / 0.75 works out at 25.78 minutes, which would put attachment at 18:34:13, and the run log has the job starting at 18:34:16.748. That felt like more than coincidence to us, but we are inferring the mechanism rather than reading it anywhere, so we would much rather hear it confirmed or corrected by someone who knows.

The gap

Job ran	18:34:16 to 19:06:17, 32.0 minutes
Session billed	18:34:13 to 20:00:00, 85.8 minutes
Attached after the job finished	53.7 minutes

The 18:00 hour is almost fully utilised, 25.78 minutes attached against 25.72 minutes running. Then the job completes at 19:06 and the session continues accruing at the flat rate until 20:00. Roughly 63% of the billed attached time was after the work was done.

Questions

Is a flat per-session rate the intended billing behaviour for an attached serverless notebook, independent of execution? Is 0.75 DBU/hr correct for PREMIUM_ALL_PURPOSE_SERVERLESS_COMPUTE_US_EAST, or is that account-specific?
Is it prorated by attached minutes within each hourly bucket? That is what the partial values and the run-log match both suggest.
Is there any way to set an idle timeout for serverless notebooks? SQL warehouses have auto-stop, classic clusters have auto-termination. I cannot find an equivalent for serverless notebooks.
Does detaching stop the session charge immediately, or does it continue through teardown?
Is any of this documented? The serverless billing docs do not mention it, and the commonly repeated claim that serverless has no idle billing appears to describe SQL warehouses rather than notebooks.

Why I am asking

For a team doing interactive development this is a material cost that is currently invisible. We did a cost estimation for our analytics platform for migration to Databricks and our platform cost estimate assumed serverless bills execution time only. However, we are now seeing that we seem to be getting much more charged than what the expected amounts were based upon how long our jobs are running.

If this behaviour is intended, I would like it confirmed so we can budget for it and tell developers what detaching actually saves them. Grateful for any pointers, and happy to share more of the raw records or the queries if that would help.

Louis_Frolio · Wednesday

Greetings @hassanashas , I did some digging and would like to share my findings.

This is one of the more carefully run experiments I've seen posted here. The timestamps against your own run log make this much easier to reason about the problem. There's also useful prior work in the thread you linked, where @Ashwin_DSA (a Databricks Delivery Solution Architect) did his own testing and confirmed that system.billing.usage is the source of truth for what you're actually charged. I'll build on that and go question by question.

Yes, an attached serverless notebook session accrues DBUs whether or not a cell is running. That part of your read is right, and it's intended behavior, not a bug. The session holds compute while attached so your next cell starts instantly, and you pay for that until it's detached or idle-terminated. You're also correct that the "serverless has no idle cost" claim floating around describes serverless SQL warehouses and serverless jobs, which spin down when the work stops. Interactive notebooks behave differently. Where I'd pump the brakes is on calling 0.75 DBU/hr a flat session rate based on one workspace's records. Your inference is reasonable, since an idle session sitting at its minimum allocation would show up as a constant rate, and an active session can bill above that because serverless autoscales. But I can't find a published baseline rate, and the number could vary by SKU, cloud, region, and account pricing. Rather than inferring it, check the sku_name on your records and join them to system.billing.list_prices to see the effective rate for your account. That turns a guess into a fact.
Yes, the metering shape you found is expected. The docs are explicit that a single workload can produce multiple records in the same timeframe, and the summed DBUs represent that hour's consumption. So hour-aligned records with prorated partials at session start and end are normal, and dividing back to attached minutes is a sound way to read them. One caution: records can take up to a day to fully land in system.billing.usage. If your session actually idle-terminated a few minutes into the 20:00 hour, a small trailing record may have arrived after you queried. Worth re-running before concluding the session ended exactly at 20:00.
There's no idle timeout knob for serverless notebooks today. Idle termination does happen automatically (the docs acknowledge it, and the session restoration feature exists specifically to soften it), but the window isn't user-configurable and its duration isn't published anywhere I can find. Your 53.7 minutes of post-job attachment is consistent with a window somewhere around an hour, but that's inference, not documentation. What you can configure is the execution timeout, which defaults to 2.5 hours. Set spark.databricks.execution.timeout in the notebook, or workspace-wide under Settings > Compute > Serverless interactive. That bounds long-running commands, though, not idle attachment, so it won't solve your problem here.
Detaching releases the session, and Ashwin's testing in the earlier thread pointed to billing stopping at teardown. That said, I can't find a public doc that guarantees detaching ends billing immediately, so I won't claim it. Since you clearly have the tooling to measure this precisely, a controlled test would settle it: attach a fresh notebook, run nothing for a fixed interval, detach at a recorded UTC time, wait for the billing data to settle, then compare the matching records. Repeat it at least once so a single hourly boundary doesn't drive the conclusion. Your billing table is the source of truth here.
Partially documented, and you've found a real gap. Idle termination is acknowledged in the notebook compute docs, and the metering granularity is covered in the serverless billing docs, but a plain public statement that an attached idle notebook accrues DBUs at a baseline rate doesn't exist as far as I can tell. If your controlled test shows post-detach DBUs, or the records won't reconcile, open a support case with your workspace, cloud, region, notebook ID, exact UTC timestamps, SKU, and redacted usage rows. And please don't post account identifiers publicly.

Now for the practical part. Your ingest notebook is a scheduled workload living on interactive compute, and that's the root of the cost surprise. I'd move it to a serverless job. A job run spins up, does the work, and terminates when the run finishes, so there's no attached-session tail to pay for. For genuinely interactive development, the guidance for your team is simple: detach when you're done, and budget for the idle window if you don't. Budget policies and alerting on system.billing.usage help make this visible.

Here's a query for your controlled test, adapted from the one Ashwin shared in the other thread, with sku_name added so you can join to list prices:

SELECT
  usage_start_time,
  usage_end_time,
  sku_name,
  cloud,
  billing_origin_product,
  usage_unit,
  usage_quantity,
  usage_metadata.notebook_id,
  usage_metadata.notebook_path,
  product_features.is_serverless
FROM system.billing.usage
WHERE usage_metadata.notebook_id = '<notebook_id>'
  AND billing_origin_product = 'INTERACTIVE'
  AND product_features.is_serverless
  AND usage_unit = 'DBU'
  AND usage_start_time >= '<start_utc>'
  AND usage_end_time <= '<end_utc>'
ORDER BY usage_start_time;

References:

Serverless compute for notebooks (execution timeout): https://learn.microsoft.com/en-us/azure/databricks/compute/serverless/notebooks
Notebook compute and idle termination: https://learn.microsoft.com/en-us/azure/databricks/notebooks/notebook-compute
Monitor the cost of serverless compute: https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/serverless-billing
Billable usage table reference: https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/billing
Serverless compute for jobs: https://learn.microsoft.com/en-us/azure/databricks/jobs/run-serverless-jobs
Earlier thread with @Ashwin_DSA's testing and billing query: https://community.databricks.com/t5/data-engineering/serverless-notebook-idle-timeout-is-it-configur...

Hope this helps, Louis

data_pulse · Wednesday

@Louis_Frolio

Thanks, this clarifies the idle billing behaviour.

I also observed a very similar pattern in another Azure workspace using the EU West serverless all-purpose SKU. The last visible notebook execution was around 09:56 UTC, while system.billing.usage continued to show INTERACTIVE / COMPUTE_TIME attributed to the same notebook until 12:00 UTC.

Interestingly, the records also included an exact 0.750000 DBU hour:

10:00–11:00 → 0.750000 DBU
11:00–12:00 → ~0.069884 DBU

So the 0.75 pattern appears in EU West as well, although it may not represent a universal baseline rate.

One additional observation: the shorter execution related records had usage_metadata.interactive_source = NOTEBOOK, while there were also overlapping records attributed to the same notebook/user where interactive_source was NULL. The NULL source records also continued after the last visible notebook execution.

Would the interactive_source = NULL records represent the attached/session level serverless consumption you described, while the NOTEBOOK records reflect the actual notebook workload activity?