@exrwebflowai
Model Serving and Routing Architecture
Hybrid setup anchored by Databricks Model Serving / Unity AI Gateway gives the best balance between deterministic latency and governance for production workloads. Use Provisioned Throughput endpoints for core, high volume production models where you must guarantee latency SLAs and fall back to Pay-per-token for bursty dev/test traffic or baseline Foundation Model APIs OpenAI, Anthropic and Gemini.
If you are orchestrating multi-step custom pipelines such as LangGraph or LangChain workflows, wrap them directly as a ResponsesAgent or ChatAgent and deploy them via Model Serving. It natively handles streaming responses, auto-scaling and tool-calling integrations with Unity Catalog Functions and AI Search (formerly Vector Search).
When external LLM APIs are required, route every outbound request through AI Gateway. It enforces centralized rate limiting, automated fallback routing, token budgeting and automatically writes request/response payloads to Unity Catalog inference tables. From an integration standpoint, your custom backend should always act as a reverse proxy to Databricks endpoints using streaming for real-time frontend UX. Do not expose Databricks serving tokens or endpoints directly to the client tier.
Vector Search and Storage Strategy
Databricks AI Search can serve as the default retriever layer unless an edge case strictly dictates otherwise. You can use Lakebase search if the data is already in Lakebase. Choose your endpoint type based on the access pattern as below
Standard Endpoints is good for real-time inference requiring quick responses. Storage-Optimized Endpoints is best for massive corpus scale 1B+ vectors where latency is acceptable but it cuts storage costs by roughly 7x.
You can check Delta Sync indexes which continuously sync against source Delta tables and trigger managed embedding transformations without dedicated ETL pipelines.
You can leverage built in managed embedding endpoints such as databricks-gte-large-en, databricks-bge-large-en, or databricks-qwen3-embedding-0-6b for multilingual corpora if required. If you need to integrate third party vector databases Pinecone, Weaviate, Milvus due to specialized index structures or constraints, encapsulate them behind a custom retriever layer or AI Gateway to prevent breaking Unity Catalog access lineage.
Observability, Drift Tracking and Quality Loops
You can structure the observability in three distinct layers to reliably find hallucinations and performance degradation
- Instrumentation & Logging - Instrument code with Tracing to capture granular spans across vector retrieval, context augmentation, model calls and guardrails. Combined with automated Inference Table logging on Model Serving, all prompt/response payloads, latency metrics and token counts land directly in queryable Delta tables within Unity Catalog.
- Evaluation & Hallucination Scoring - Schedule evaluation jobs against sampled production traces. Built-in metrics like Retrieval Grounded ness measuring context hallucination and Correctness allow you to track drift systematically over time. To tailor these evaluations to your domain, use MemAlign to align LLM judges with internal expert evaluations.
- Automated Optimization & Alerting - You can use Databricks SQL Alerts on top of the inference tables to trigger notifications via Slack, email or PagerDuty when hallucination rates, error counts or p99 latencies exceed thresholds. For continuous improvement, feed evaluation failures into GEPA optimize_prompts to iteratively refine system prompts against the benchmark suites.
The core design principle is maintaining the full lifecycle - data preparation, embeddings, vector retrieval, serving, logging and evaluation inside the Unity Catalog governance boundary. It eliminates cross cloud egress latency, simplifies data lineage and enforces unified UC access control policies across data and AI assets.
