How is Serverless Compute implemented under the hood?

[Darshanrawal971]

Hi everyone,

I've been reading about Serverless Compute and have a question about the underlying architecture.
In the classic compute model, clusters are provisioned in the customer's cloud account. With Serverless, Databricks manages the compute, but I'm trying to understand what that actually means under the hood.

Is Serverless still running on Azure/AWS/GCP VMs that Databricks manages in its own cloud account, or is there a different execution model behind it? Also, how is compute isolated across tenants while keeping startup times so low?

I couldn't find much detail beyond the high-level architecture in the documentation. If anyone knows of an engineering blog, Summit session, or technical deep dive that explains this, I'd appreciate the reference.

Thanks!

[balajij8]

Hi Darshan,

Serverless runs on the underlying cloud provider's VMs but the fundamental shift is where those resources live and who manages their lifecycle.

In the classic compute model, Databricks orchestrates resources directly inside your cloud account. With Serverless, Databricks moves the compute layer into a Databricks-managed serverless compute plane. This plane operates alongside your classic compute setup but sits entirely within Databricks' own cloud accounts. It remains strictly constrained to the same cloud region as your workspace to minimize latency and avoid cross-region egress.

To isolate tenants safely on this managed infrastructure Databricks relies on several layers of network and compute boundaries

• Workspace-level perimeters: Each workspace's serverless compute is logically isolated within its own dedicated network boundary.

• Cluster-to-cluster controls: Additional network controls prevent lateral movement between different clusters, even those owned by the same customer. Serverless compute resources are generally not assigned public IP addresses.

• Backbone routing: All traffic between the Databricks control plane and the serverless compute plane routes exclusively over the cloud provider’s private backbone network.

The reason they can achieve near-zero cluster startup times is essentially predictive capacity management. Databricks maintains warm pools of pre-provisioned compute resources behind the scenes. When you execute a command in performance-optimized mode, Databricks dynamically attaches these pre-warmed instances to your workspace's isolated network in a matter of seconds.

More details here

[ShamenParis]

Hi @Darshanrawal971 

Great question! To understand how Serverless works under the hood, it helps to break it down into two main concepts: Warm Pools and Secure Networking.

1. How Databricks manages the Compute (Warm Pools)

Instead of spinning up new Virtual Machines (VMs) in your cloud account every time you run a job, Databricks runs the compute in its own managed cloud account. To make sure things start instantly, Databricks keeps a "warm pool" of VMs already running and waiting.

When you submit a query, Databricks grabs a VM from this warm pool and assigns it to your workload. Once your job is done, that compute is wiped—it never goes back into the pool. This gives you the instant startup time without sacrificing isolation.

2. How it accesses your data (NCC)

Because the compute is running in Databricks' cloud account and your data is sitting in your cloud account, they need a secure way to talk to each other. This is where Network Connectivity Configurations (NCC) come in.

NCC uses private cloud networks to route traffic securely. This means your data never travels over the public internet; it stays entirely on the cloud provider's private backbone to securely grant access from the Serverless compute plane to your data.

Architecture Sketch (I made this for your reference )

References to dive deeper:

If you want to read more about this, check out the official documentation:

• High-Level Architecture: Databricks AWS Architecture Docs

• Serverless Compute Networking: Serverless Network Security

Hope this helps clear it up!