Taking the proven MLOps deploy-code pattern and adapting it for GenAI agents: evaluation-driven CI/CD, a native observability loop, and open OpenTelemetry integration.
From MLOps to AgentOps
Most data teams already have a mental model for shipping ML to production: the "deploy-code" reference architecture from the Big Book of MLOps. One repository, three branches mapping to three environments (dev, staging, prod), Git-driven CI/CD, MLflow for tracking, Unity Catalog for governance, and a champion/challenger gate before anything reaches production.
When teams start building AI agents (RAG chatbots, tool-calling agents, multi-agent systems), the natural question is: what does that picture look like for agents? The backbone carries over almost entirely. What changes is why. Agents are probabilistic, so the rigor moves from a one-time accuracy check to a continuous loop: you measure quality with evaluation, gate promotion on it, trace every request in production, and feed what you learn back into the next iteration.
Picture a customer-support agent that uses Vector Search for retrieval and Unity Catalog Functions to look up tickets. In dev you evaluate it against a curated set; in staging it has to clear evaluation thresholds before it is promoted; in production, its traces and user feedback become new evaluation cases. Operating the agent means operating that loop.
The reference flow
Read it top to bottom. A Git provider drives three workspaces (Development, Staging, Production), all sitting on a single Unity Catalog.
Figure 1 — Development workspace: the dev inner loop, with a tight trace → curate → re-evaluate cycle before promotion.
Development workspace (inner loop). A developer branches off dev and works against the Dev catalog. A Data Preparation job (ingestion, preprocessing, Vector Search indexing) builds the retrieval layer. An Agent Development job assembles the agent from a tool library (Unity Catalog functions, MCP servers, and Skills the agent can call) and runs agent evaluation with mlflow.genai.evaluate() and LLM judges/scorers on a curated dataset. This is where human judgment enters the loop: subject-matter experts label and review examples to build that curated dataset, and the LLM judges are defined and then aligned against those human labels so an automated score means what an expert would say. Every run logs traces and evaluation results to the MLflow tracking server (Experiments, Traces, Prompt Registry). The agent is then deployed, increasingly as a Databricks App, and exercised through a chat/review interface - where reviewers can add feedback and labels that flow straight back into the curated dataset, keeping the scorers aligned as the agent evolves.
main to Staging (automated testing). A pull request to main triggers unit tests; on merge, the Staging workspace runs integration tests (do the tools and the assembled agent work end to end) plus the key AgentOps addition: an evaluation gate. The agent is scored on a larger eval set this is the agent-world analog of champion/challenger except you’re gating on evaluation of a composed system, not a single accuracy metric. A version is promoted only when it clears the defined thresholds for every scorer in the evaluation.
release to Production (continuous deployment). Merging to release triggers continuous deployment: the agent is registered in Unity Catalog and deployed to production, serving both batch inference and live traffic.
What's genuinely different for agents
- Deploy-code beats deploy-model, even more than in classic ML. An agent is mostly code, prompts, and tools, often with no training step, so there is no trained-weights artifact to promote. You promote the recipe across environments and rebuild against each environment's catalog. Prompts are the one component that follows an artifact-style promotion: versioned Unity Catalog assets moved by alias. This is also why Databricks Apps fit so well: they are git-versioned and CI/CD-native. (Model Serving with the ResponsesAgent flavor remains fully supported; Apps is the recommended path for new agents.) In practice the promotable recipe is a Databricks Asset Bundle (databricks.yml): one bundle, deployed per target with databricks bundle deploy -t dev / staging / prod, so CI/CD rebuilds jobs, the app, and serving config against each environment's catalog.
- The gate is evaluation, not accuracy. You cannot assert output == expected on a non-deterministic agent. So evaluation, using LLM-as-judge and programmatic scorers against curated datasets, becomes the unit of truth and a first-class CI gate.
- Observability is not optional. Because behavior is emergent, production traffic is part of the specification. You trace everything and monitor continuously.
- Governance gets harder, not just bigger. In classic ML you govern who can reach a model or a dataset. An agent adds a live question: what is it allowed to do? It calls tools, retrieves data, and takes actions at runtime, so governance has to cover its operating scope and context, which tools and data it may touch, under what policies, not just static asset access. On Databricks that means Unity Catalog governing tools, prompts, traces, and evaluation datasets under one lineage and access model, enforced through context and access policies and guardrails, with a human on the loop setting the boundaries rather than in it approving every step.
The observability loop
This is the heart of AgentOps:
- MLflow 3 Tracing captures every step of every request: LLM calls, tool calls, retrieval, latency, cost. Tracing is deployment-agnostic: it works whether the agent runs in a Databricks App, a job, or outside Databricks.
- Traces land in Unity Catalog as governed Delta tables — queryable with SQL, Genie, and AI/BI, and PII-maskable. Unity AI Gateway (Beta) payload logging captures request/response into UC alongside them. Note: an MLflow experiment caps at 100,000 traces by default; for higher volume, store traces in Unity Catalog to lift the cap and get long-term retention. See: Store OpenTelemetry traces in Unity Catalog.
- Production monitoring runs the same scorers and LLM judges online on production traffic (quality, correctness, safety) alongside operational metrics.
- Human feedback (a review/labeling app) flows back into evaluation datasets, so today's production failure becomes tomorrow's test case. The loop, not any single deploy, is the product.
- Because MLflow traces are OpenTelemetry-native, Unity Catalog stays the system of record while you can optionally dual-export the same trace telemetry to external telemetry/observability tools you already run, such as Datadog or Grafana, for unified infra/APM.
The Databricks AgentOps methodology
Databricks gives you an end-to-end, native methodology for the AgentOps loop, with no assembly required:
- Trace: MLflow 3 captures every step of every request.
- Evaluate: mlflow.genai.evaluate() with LLM-as-judge and programmatic scorers against curated datasets in Unity Catalog.
- Version prompts: the MLflow Prompt Registry, promoted by alias.
- Monitor: Production monitoring runs the same scorers online.
- Govern: Unity Catalog governs all of it (traces, prompts, evaluation datasets, tools, and models) with one lineage and access model. For agents this goes beyond who can reach a model or dataset to what an agent may do, its operating scope and context, enforced through context and access policies, and guardrails.
Governance belongs at the core of agentic development. Because behavior is non-deterministic, you define an agent's scope and context up front and check it continuously, keeping a human on the loop, setting the boundaries, rather than in it, approving every step.
Because the whole loop lives on one platform, evaluation results, traces, and governance are joined by default rather than stitched together. The methodology is also open: MLflow tracing is OpenTelemetry-native, so the same telemetry can flow to the observability tools you already run, while Unity Catalog stays the system of record.
Two practices keep the loop coherent: make Unity Catalog the system of record for traces, prompts, and evaluation datasets, and keep one source of truth for evaluations so the staging gate calls a single authority.
Why this architecture works well on Databricks
An agent is only as good as the data, governance, and serving around it, and that is where a lakehouse helps:
- One governed platform for data and AI. Unity Catalog governs tables, models, functions, vector indexes, prompts, traces, and evaluation datasets together, with one lineage and access model.
- Retrieval and tools live next to the data. Vector Search and Unity Catalog Functions (as agent tools) are native, with no data movement and no separate vector database to secure and govern.
- Flexible, git-native deployment. Ship agents as Databricks Apps (git-versioned and CI/CD-native) or on Model Serving, with deployment-agnostic tracing either way.
- Enterprise-grade by default. Per-environment workspace and catalog isolation, PII masking, fine-grained access control, and end-to-end lineage.
The bottom line
Agents aren't harder to operate because they're magical; they're harder because they're probabilistic. The answer is not less rigor, but different rigor: evaluation gates, full tracing, and a closed feedback loop. That is the real shift from MLOps to AgentOps.
