Databricks AI Model Serving in production: scaling... - Databricks Community

srikantdas

Getting a model behind a REST endpoint is the easy part; however keeping it fast, affordable, and stable once production traffic shows up is where teams spend their real time.

Databricks Model Serving (formerly called Mosaic AI Model Serving) is a serverless service that turns any model into an autoscaling REST endpoint and manages the infrastructure for you, but “managed” does not mean automatically optimize for your specific workload, rather it will manage your infrastructure setup, maintenance etc. while you still can make the choices during setup time and decide whether you end up paying for idle GPUs, watching p99 latency spike during a traffic burst, or quietly doing both. With Unity AI Gateway, you govern models from any source, Databricks-hosted and third-party or external, behind one layer for guardrails, rate limiting, cost tracking, and observability, without locking into a single provider, Model Serving is the serving layer in that bigger picture, a single platform that runs classic ML, deep learning, and today's GenAI and LLMs, including the models behind your agents and MCP tools, all under one governance layer with Unity Catalog and Unity AI Gateway.

This post walks through how to set Model Serving up for production around three levers: how you scale, what you pay, and how fast you respond. When you reduce the latency, the cost tends to climb and, when you reduce cost, the tail latency tends to suffer. None of the three is free but the goal is to know which one matters most for the workload in front of you, and tune deliberately.

01 Start with the right endpoint type

Your initial choice of endpoint dictates all downstream cost and latency outcomes. Model Serving provides four distinct endpoint types, each with unique performance characteristics and cost profiles.

Custom model endpoint for your own MLflow models promoted from Unity Catalog, on CPU or GPU. These are your classifiers, recommenders, embeddings, and anything you trained or fine-tuned yourself.
Foundation Model API, provisioned throughput for open or fine-tuned LLMs that need dedicated, guaranteed capacity. Databricks recommends this mode for production GenAI applications/workloads. This includes support for Partner models (OpenAI, Anthropic, Gemini), Open models (Llama, GPT OSS, Kimi, Qwen), and External models (third party models hosted outside of Databricks).
Foundation Model API, pay-per-token for the same models on a shared, multi-tenant endpoint. Cheap to start, but latency varies under load, so it suits experimentation and low volume.
External model endpoint for third-party providers such as OpenAI or Anthropic, fronted by Databricks so you keep one governance layer across everything.

Figure 1. Pick the endpoint shape from what you are serving and where it runs. Everything downstream → autoscaling, billing, and latency tuning → follows from this branch.

Two scaling terms appear throughout this section. Scale to zero means an endpoint spins its compute all the way down to zero when idle, so you pay nothing while it is idle, at the cost of a cold-start wake-up on the next request. Provisioned concurrency keeps a warm floor of replicas always running, so there is no cold start.

Endpoint type	Best for	How it scales	How you pay
Custom · CPU	Classic ML, embeddings, high-QPS real-time	Provisioned concurrency; scale-to-zero in dev	DBU per second of active compute
Custom · GPU	Deep learning, vision, self-hosted models, fine tuned OSS models	Provisioned concurrency; GPU autoscaling is slower	DBU per second, by GPU tier
Provisioned throughput	Production LLM and GenAI apps	Autoscales within a tokens-per-second range	Per hour, by the throughput range
Pay-per-token	Experiments, low or bursty volume	Shared multi-tenant; latency varies under load	Per million input and output tokens
External model	Third-party APIs under your governance	Delegated to the provider	Provider token fees, billed directly

02 The production reference architecture

A production endpoint is rarely just an endpoint. In practice it is wrapped as a governance layer on the request path into the model, and an observability layer on the telemetry path coming out of it, all sitting on Unity Catalog. The shape below is what we recommend teams converge on.

Figure 2. Requests flow left to right through the Gateway into the endpoint; telemetry flows down into Delta-backed inference tables and monitoring. Unity Catalog is the substrate for the model registry, permissions, and usage system tables.

03 Scaling: the autoscaling decisions that matter

Scale to zero is suitable for Development environments

Scale to zero spins compute down to nothing when an endpoint is idle, which is excellent for keeping dev and test inexpensive. It is not a production setting: a scaled-to-zero endpoint has no guaranteed capacity, and the first request after an idle period is slow because it triggers a cold start; the wake-up can add seconds while compute spins back up. On GPUs, this is worse as GPU autoscaling takes even longer. For anything user-facing or latency-sensitive, turn off scale to zero and keep a warm floor of capacity.

Provisioned concurrency: size it from data

Provisioned concurrency is the maximum number of parallel requests an endpoint will handle, and it is the single most useful knob for real-time models. There is a clean formula to start from:

SIZING RULE provisioned concurrency = peak QPS (Query Per Second) × model execution time (seconds)

Set the minimum high enough to absorb baseline traffic without queuing, and the maximum high enough to ride out spikes while still capping spend. The honest way to find both numbers is a load test with a representative payload, because execution time is always hard to guess. If you start seeing HTTP 429 responses under load, that is your signal that requests are queuing and concurrency is too low. A standard endpoint has a limit of 200 requests per second, and route optimization takes that up to 300,000. Requests at once. Separately, provisioned concurrency (how many requests run at once) is 4,096 per workspace by default, and a single model can reach 1,024 with a custom endpoint and route optimization. All of these can be raised on request.

Provisioned throughput for LLMs

For foundation models in production, provisioned throughput reserves dedicated capacity and is configured as a range of tokens per second (newer model families use model units instead). The endpoint autoscales within that range, and you are billed per hour for the range it is scaled to.Therefore, you pay for extra capacity only while you are using it. Because output tokens are more expensive to generate than input tokens, the right range depends on your real prompt and response shapes. Benchmark with a representative dataset rather than reading throughput off a spec sheet.

Route optimization for high throughput

A standard endpoint tops out at 200 QPS per workspace. Route optimization improves the network path between client and model and lifts that ceiling: published limits move to 50,000+ QPS, and Databricks reports real-time workloads scaling to 300K+ QPS on CPU. Opt for it when you need more than 200 QPS or want to push the per-request overhead toward sub-50ms. Two caveats: route optimization is available for custom model endpoints only (not Foundation Model APIs or external models), and it requires OAuth tokens rather than personal access tokens.To enable it, select Enable route optimization when creating the endpoint (it can't be added later) and query the route-optimized URL with an OAuth token, see Route optimization on serving endpoints.

04 Cost

Model Serving is serverless and billed in DBUs per second of active compute, and because it is serverless the DBU rate already includes the underlying cloud instance, so you see a single charge rather than a separate VM bill. What you pay comes down to how many DBUs each instance uses per hour, as published on the Databricks pricing page:

Instance size	GPU configuration	DBUs / hour
Small	T4 or equivalent	10.48
Medium	A10G × 1	20.00
Medium 4X	A10G × 4	112.00
Medium 8X	A10G × 8	290.80
Large 8X 40GB	A100 40GB × 8	538.40
Large 8X 80GB	A100 80GB × 8	628.00

DBU rates from the Model Serving pricing page. Rates vary by cloud, region, and tier and change over time, so confirm current figures there before building a cost model.

The takeaway is simple: a Small T4 endpoint costs little to keep warm, but a large multi-GPU endpoint left running around the clock can use hundreds of DBUs an hour (up to 628 on a Large 8X 80GB, per the table above). For LLMs the pricing model differs again, pay-per-token is billed per million input and output tokens, while provisioned throughput is billed per hour by its configured range. Practical levers for keeping the bill honest:

Start on CPU. Move a custom model to GPU only when you measure high inference latency or the model genuinely needs specialized compute. Do not default to GPU.
Right-size the GPU tier after a load test. The jump from a single A10G (20 DBUs/hour) to a Large 8X 80GB (628) is more than a 30x difference in spend.
Shrink the model. Quantization or pruning cuts both latency and the compute you pay for, and Databricks applies its own LLM serving optimizations on top.
Use scale to zero where cold starts are acceptable (dev, internal tools, spiky low-volume endpoints) and keep production warm.
Find the waste. Usage system tables log every request with requester and cost detail, so you can attribute spend by team and shut down idle endpoints.

05 Latency

When a response feels slow, the time is hiding in one of a few places. Knowing which segment is costing you points straight at the lever that fixes it.

Figure 3. Segment widths are illustrative. Inference is the real work; queue and cold start are the avoidable tax. Each segment maps to a specific lever.

The cost versus latency tradeoff

Most latency levers cost more money, and most cost levers increase latency.

Setup	Idle cost	Cold start	Tail latency	Use when
Scale to zero	none	yes	spiky	Dev, internal tools, low volume
Provisioned concurrency, min > 0	always on	none	low	Production real-time models
Provisioned + route optimization	highest	none	lowest	High QPS, sub-50ms overhead
Provisioned throughput (LLM)	always on	none	consistent	Customer-facing GenAI

Other levers are close to free latency wins: enable client-side batching when your app can group requests, reuse connection pooling (the Databricks SDK does this for you), and offload heavy pre- and post-processing off the serving path so the endpoint only does inference.

06 Govern and observe with Unity AI Gateway

Unity AI Gateway (the governance layer that was Mosaic AI Gateway) is what keeps a production endpoint controllable. Configure it before launch, not after the first incident.

Rate limiting in queries per minute and tokens per minute, set on the endpoint or per user, group, or service principal. This is your first line of defense against a runaway client.
AI Guardrails to block or mask unsafe content and PII on the way in and out, set at endpoint or request level.
Payload logging into inference tables, which continuously capture raw inputs, outputs, HTTP status codes, and latency as a Delta table in Unity Catalog. That table feeds Lakehouse Monitoring for quality and drift, and doubles as an evaluation or fine-tuning corpus.
Usage tracking into system tables for cost attribution across every endpoint in the account.
Fallbacks and traffic splitting for resilience and safe rollouts: fallbacks (on external endpoints) keep you serving during a provider outage, and traffic splitting lets you carry a new model version before shifting all traffic.

WATCH THIS LIMIT Turning on inference tables caps total max concurrency on a single endpoint at 128. If you are pushing very high concurrency and need full payload logging, plan for it or ask your account team to raise the limit.

07 The usual failure pattern, and the fix

To make the levers concrete, here is the pattern we see most often when a custom model moves from notebook to production. It is an illustrative walkthrough rather than one customer’s story, so it carries no invented latency or traffic figures. Every limit and lever below is documented and linked.

What goes wrong

A team ships a custom CPU endpoint on the defaults: scale to zero left on from dev, modest concurrency, a standard rather than route-optimized endpoint, and clients on personal access tokens. A smooth, uniform load test looks fine, and then real traffic bites in three places at once. A standard endpoint is capped at 200 QPS per workspace, so a spike beyond it is throttled with HTTP 429 errors. Once requests queue, tail latency climbs. And 429s under load are the documented sign that concurrency is set too low. And the first calls after an idle window pay a wake-up, because scale to zero is not a production setting.

The fix is configuration, not hardware

Throwing a GPU at a routing problem is the classic wrong turn. A single A10G lists at 20 DBUs/hour and an eight-GPU box at 290.8, and neither touches a QPS ceiling. The fixes that do work are all configuration:

Size concurrency from the documented formula (provisioned concurrency = peak QPS × execution time), measured under load rather than guessed.
Disable scale to zero and keep a warm minimum so no request pays the cold-start tax.
Turn on route optimization to lift the ceiling from 200 to 50,000+ QPS (Databricks reports 300K+ QPS on CPU). It is custom-endpoint-only and needs OAuth, not PATs.
Reuse connection pooling and client-side batching to cut per-request overhead.

THE TRADEOFF TO KNOW UP FRONT Route optimization is not free of tradeoffs. On a custom endpoint, route-optimized endpoints do not currently support rate limiting or usage tracking, and inference-table logging for them is still in preview. If you need both maximum throughput and full gateway governance, decide where each one lives before launch.

Symptom	What it usually means	Lever
HTTP 429 under load	Past the standard-endpoint QPS ceiling	Route optimization, 200 → 50,000+ QPS
Tail latency climbs under load	Concurrency too low; requests queue	Size from the QPS × execution-time formula
Slow first request after idle	Scale-to-zero cold start	Disable scale to zero, keep a warm minimum

The lesson is, measure execution time, size concurrency from it, keep a warm floor, and clear the routing ceiling. Also consider what route optimization costs you on the governance side.

08 Two more shapes, in brief

Not every workload has a high-QPS requirement but there are also two more patterns we often reach for:

CUSTOMER-FACING RAG / LLM

An assistant your users talk to

Quality, safety, and steady latency matter more than saving idle cost.

Provisioned throughput, not pay-per-token. Reserve dedicated capacity sized to your measured peak, with the minimum above zero so it never cold-starts.
Budget for guardrails. PII masking and toxicity filters run their own moderation pass and add latency. It is worth it, but put it in the latency budget rather than discovering it in production.
Rate-limit per team in tokens-per-minute through Unity AI Gateway, so one noisy client cannot drain the shared budget.
Fallbacks and traffic splitting for safe rollouts; mind the 4 MB payload limit on foundation endpoints.

EXPERIMENTATION / INTERNAL TOOLS

Spiky, low-volume, cost-sensitive

Here, paying for idle capacity is the mistake, not the occasional cold start.

Pay-per-token, or CPU with scale to zero on. Accept the first-request wake-up to keep idle spend at zero.
Cap spends at the gateway with queries- and tokens-per-minute limits per user or group, before someone’s notebook loop runs up a bill.
Promote on evidence. Use the usage system tables to see which experiments actually get traffic, and move only those to provisioned capacity.

09 The pre-production checklist

Load test with representative payloads. Derive concurrency and throughput from measured execution time, not estimates.

Disable scale to zero for anything user-facing, and keep a warm minimum.

Turn on route optimization for custom models above 200 QPS or with low-overhead requirements, and move clients to OAuth.

Enable inference tables and usage tracking, build a dashboard, and alert on 429s, p95 and p99 latency, and error rate.

Set rate limits and guardrails with Unity AI Gateway before launch.

Plan fallbacks and a rollback path, and use traffic splitting for canary releases.

Right-size compute. Start on CPU, move to GPU only when the numbers justify it, and revisit the GPU tier after testing.

It is the difference between an endpoint that demos well and one that survives a Monday morning traffic spike at a price you can defend. Pick the endpoint type from your workload, keep production capacity warm, size scaling from real measurements, and let Unity Catalog and Unity AI Gateway give you the governance and visibility to keep tuning. Decide which lever your workload cares about most and then make the other two earn their cost.

Databricks AI Model Serving in production: scaling, cost, and latency lessons