Sharing the two-column version I use when someone asks which compute to pick.
Classic: the control plane requests VMs from your cloud provider and the cluster runs in your own account and network. You own instance types, autoscaling and networking, and you pay the cloud provider for the machines plus Databricks for the DBUs.
Serverless: resources run in the serverless compute plane, managed by Databricks, and nothing is provisioned in your cloud account. Tables stay in your own storage.
Before moving an existing job I check it against the documented serverless limitations: R is not supported; compute-scoped libraries and init scripts are not supported; only Spark Connect APIs are available, so RDD APIs are out; Trigger.AvailableNow() is supported but Trigger.Continuous and Trigger.ProcessingTime are not; and there is a 7-day maximum runtime after which a run is terminated and not retried.
Animated diagram of both paths, 2:36: https://youtu.be/s8452E2Y6WM
The series in order: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4
Interested in how others decide the boundary in a mixed estate - per workload, per team, or per environment?
