Building an LLM-powered prototype is easy. A few lines of code, a hosted model, and you have an agent that classifies tickets, summarizes documents, or answers questions. The hard part comes after: measuring its quality, promoting only the versions you trust, deploying it the same way every time, and being able to show an auditor what it did in production.
Many practitioners treat that operational layer as a chore. But once an LLM application touches real users, the organization needs what it has always needed from software: reproducibility, quality gates, controlled releases, and governance. That is what LLMOps gives you, and on Databricks, it stays light.
This quickstart carries one small application through the whole lifecycle. The LLMOps Quickstart repository implements a customer support ticket classifier. Given the free text of a ticket, it returns one of five categories: billing, technical_issue, feature_request, account_management, or other. The classifier travels from raw data to a governed, running agent:
Data ingestion → agent build → evaluation → approval → deployment → inference
The companion MLOps Quickstart follows the same shape for classic ML. This one is its LLM sibling, built on the latest tooling: an agent served as a Databricks App, a Unity AI Gateway model service for the LLM, and MLflow 3 GenAI evaluation.
In this guide, we'll build a simple LLM-based classifier for support tickets. This is a common scenario for support teams, like a help-desk or IT. Then, we'll deploy that agent, using Databricks Apps, after confirming its quality using MLFlow's native tooling. All of this is then wrapped into a Declarative Automation Bundle (DAB) for easy redeployment into a production environment.
What you should know first
The quickstart assumes you are comfortable with Python and the command line, the basics of Unity Catalog (catalogs, schemas, tables, grants), and running the Databricks CLI. You do not need prior MLflow, agent, or Declarative Automation Bundles experience. Each is introduced as you reach it. For a deeper grounding, see the Databricks Academy courses DevOps Essentials for Data Engineering (CI/CD and bundles) and Building Agentic Applications on Databricks (agents, MLflow tracing, and evaluation).
What you'll need
- The Databricks CLI and uv
- A workspace with Unity Catalog, Foundation Model APIs, Databricks Apps, and Unity AI Gateway model services enabled
- Unity Catalog privileges for the identity the jobs and app run as. The jobs run on serverless compute, and its runtime identity needs the catalog and schema grants.
- A model service for the LLM (more on this below)
Step 1: Create a model service for the LLM
A Unity AI Gateway model service is a Unity Catalog securable that represents a governed LLM endpoint. The agent references it by its fully-qualified name, such as qs_catalog.default.claude-sonnet-5, and the AI Gateway routes the call. Because the service is a UC object, its access control, rate limits, and payload logging live in Unity Catalog rather than in the application. Governing the foundation model as a UC securable is the point: the same permission and audit model you use for tables now covers model access.
During the model services beta you create the service once in the AI Gateway UI. Code creation is not available yet, so this is a one-time manual step. Create one for the model you want the agent to use.
Step 2: Clone and configure
$ git clone https://github.com/databricks-solutions/databricks-blogposts.git
$ cd databricks-blogposts/2026-06-llmops-quickstartSettings are bundle variables with sensible defaults:
|
Variable |
Default |
Description |
|
catalog_name |
main |
Unity Catalog catalog (must already exist) |
|
schema_name |
llmops_quickstart |
UC schema (created for you) |
|
llm_model |
main.default.claude-sonnet-5 |
Fully-qualified name of the model service the agent calls |
'main' is a common catalog name, so you may already have one. To keep the quickstart self-contained and aligned with the MLOps Quickstart, point it at a dedicated catalog such as qs_catalog. If you don't want to modify the files, you can override at deploy time using '--var':
$ databricks bundle deploy \
--var="catalog_name=qs_catalog" \
--var="llm_model=qs_catalog.default.claude-sonnet-5"Step 3: Deploy the bundle
$ databricks bundle deploy --var="catalog_name=qs_catalog" --var="llm_model=qs_catalog.default.claude-sonnet-5"A bundle (Declarative Automation Bundles, or DABs) is a folder of YAML plus the notebooks and other files its jobs and apps need. deploy creates the schema, the MLflow experiment, the data-ingestion job, and the app. A separate prod target deploys the same setup against its own schema, and the same commands drop into a GitHub Actions or Azure DevOps pipeline.
Step 4: Ingest the data
$ databricks bundle run data_preprocessing_job --var="catalog_name=qs_catalog"This writes 30 hand-labelled support tickets, six per category, to a Unity Catalog managed table named support_tickets. The set is deliberately small enough to read in one screen, and it doubles as the evaluation data. When you adapt the quickstart, this is the notebook you replace with your own data.
Step 5: The agent
The agent is a Databricks App: a small FastAPI server built on MLflow's GenAI agent server, with a single @invoke handler that takes a ticket and returns a category. A few things are worth calling out.
The agent calls the LLM through the model service from Step 1. The call goes to the AI Gateway, which enforces governance and logs the request. The model service name comes from one environment variable, LLM_MODEL, so switching models is a one-line change, for example to an open model like qs_catalog.default.gpt-oss-120b.
The handler is deliberately simple: build the prompt, call the model, return the category. One detail saves confusion when you swap in a reasoning model such as Claude Sonnet 5 or a GPT-5 variant. Those models can return their answer as a list of typed content blocks, a reasoning block followed by a text block, rather than a plain string. The agent concatenates the text blocks, so it works with both reasoning and non-reasoning models.
Serving the agent as an app is the current recommendation for most agents. Model Serving with agents.deploy() still exists for special, custom cases, but an app gives you a first-class deployment surface, a service principal identity, and a place to add a UI later.
Step 6: Evaluate
Evaluation is run as a uv task, this is important because these exact commands can be used on your CI/CD pipeline later.
$ uv sync
$ uv run agent-evaluate
Evaluation uses mlflow.genai.evaluate, MLflow 3's tooling for GenAI, to run the agent over all 30 tickets with two scorers. The first, exact_match, is a small deterministic scorer: the predicted category must equal the labelled one. For a fixed set of classes this is the honest quality metric, and it is the gate. The second is the out-of-the-box Correctness LLM judge, included to show what MLflow's GenAI evaluation offers. It runs alongside but does not gate promotion. The Databricks Academy course Building Agentic Applications on Databricks covers agent evaluation in depth.
Every prediction is captured as an MLflow Trace, so the evaluation run gives you a per-ticket table of inputs, outputs, expectations, and scores. That is the difference between knowing accuracy was 90% and being able to open the three tickets it missed and see why.
Note that agent-evaluate exits with an error if exact-match accuracy is below the threshold, which defaults to 80%. That makes it a CI gate: a model that is not good enough stops the pipeline.
Step 7: Approve and deploy
Evaluation is an automated process, but human approval is the gold-standard for any GenAI application. Once you have reviewed the evaluation run and approved the model, deploy the app:
$ databricks apps deploy llmops-quickstart-classifier \
--source-code-path "/Workspace/Users/<your-email-address>/.bundle/llmops-quickstart/dev/files"Nothing reaches production until that approval. In a regulated setting the human step is often the whole point; here it is one deliberate command between a passing evaluation and live traffic.
The deployed app exposes the classifier at /invocations. Every request to the LLM goes through the model service, so the AI Gateway governs and logs it. That log, in Unity Catalog, is your audit trail and the raw material for monitoring.
Step 8: Inference
Send a ticket to the running app:
import requests
from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
app = w.apps.get("llmops-quickstart-classifier-dev")
resp = requests.post(
f"{app.url}/invocations",
headers={"Authorization": f"Bearer {w.config.oauth_token().access_token}"},
json={"ticket": "I was billed twice for my annual plan."},
timeout=60,
)
print(resp.json()["category"]) # billing
For batch scoring, read support_tickets and call the app for each row.
Governance recap
Working through the steps, you end up with a governed system without too much effort. The LLM is a Unity Catalog securable, so access, rate limits, and payload logging are managed in UC. The AI Gateway logs every request the agent makes. Evaluation is a quality gate in code, and a human approves before anything ships. A natural next step is data profiling (formerly Lakehouse Monitoring) on the logged traffic, for quality and drift tracking over time.
Before you call it done
- `databricks bundle validate` passes
- The ingestion job wrote support_tickets
- `uv run agent-evaluate` clears the threshold, with traces in the experiment
- The app is deployed and /invocations returns a category
- The model service shows the agent's traffic
- The prod target deploys into its own schema
Wrapping up
The point of the quickstart is not the ticket classifier. It is the skeleton around it. Once you have an agent served as an app, an LLM call governed through a model service, a GenAI evaluation gate, and a human approval step, you can drop in your own use case and keep the operational backbone.
LLMOps does not have to be the bureaucratic part of the job. Databricks offers a wide suite of tools to facilitate deployment and development. The repo is public and runs end to end. Clone it, try it in your own workspace, and open an issue if you have questions.
DISCLAIMER: This code is provided as-is, for educational purposes, with best effort support, and not maintained for Production, aiming to be a reference for development. If you run into problems, open an issue on the GitHub repository.
