https://community.databricks.com/t5/technical-blog/evaluate-multi-turn-chatbots-on-databricks-with-mlflow-a-step-by/ba-p/114741 <P><SPAN>Hi, I’m Debu. I spend a lot of my day building and stress‑testing LLM‑powered systems, and one lesson keeps coming back: </SPAN><STRONG>if you don’t measure your agent’s behavior over an entire conversation, you’re flying blind</STRONG><SPAN>. Below is the exact notebook pattern I use on Databricks to score a chatbot’s performance turn‑by‑turn and track those numbers over time.</SPAN></P> <H2><STRONG>Why Multi‑Turn Evaluation?</STRONG></H2> <P><SPAN>Real users don’t stop after one message. They change topics, ask follow‑ups, and expect the bot to remember context. A single‑turn test can’t surface issues like:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>losing track of earlier instructions,</SPAN><SPAN><BR /><BR /></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>contradicting itself three turns later, or</SPAN><SPAN><BR /><BR /></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>drifting into unsafe territory when the dialog gets longer.</SPAN><SPAN><BR /><BR /></SPAN></LI> </UL> <P><SPAN>That’s why every example you’ll see here treats the conversation as a list of messages—not isolated prompts.</SPAN></P> <H2><STRONG>What You’ll Build</STRONG></H2> <OL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Set up the notebook environment</STRONG><SPAN> with the </SPAN><SPAN>databricks-sdk</SPAN><SPAN> and </SPAN><SPAN>databricks-agents</SPAN><SPAN> libraries.</SPAN><SPAN><BR /><BR /></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Create a small multi‑turn eval set</STRONG><SPAN>—two dialogs, each with its own rubric.</SPAN><SPAN><BR /><BR /></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Write a tiny rule‑based agent</STRONG><SPAN> (swap it for your real model later) and wrap it in </SPAN><SPAN>mlflow.trace</SPAN><SPAN>.</SPAN><SPAN><BR /><BR /></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Define global guidelines</STRONG><SPAN> for helpfulness, clarity, and safety.</SPAN><SPAN><BR /><BR /></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Run </STRONG><STRONG>mlflow.evaluate</STRONG><SPAN> with the built‑in Databricks agent grader.</SPAN><SPAN><BR /><BR /></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Store the scores in Delta</STRONG><SPAN> so you can watch trends and catch regressions.</SPAN><SPAN><BR /><BR /></SPAN></LI> </OL> <H2><STRONG>1  — Prerequisites &amp; Environment Setup</STRONG></H2> <P><SPAN>I’m using DBR 14.3 LTS and MLflow ≥ 2.12. Install the extras and restart Python so Databricks picks them up:</SPAN></P> <LI-CODE lang="python">%pip install databricks-sdk databricks-agents dbutils.library.restartPython() </LI-CODE> <P><SPAN>Then pull in the usual suspects:</SPAN></P> <LI-CODE lang="python">import mlflow from mlflow.deployments import get_deploy_client # optional for prod deploys import pandas as pd</LI-CODE> <H2><STRONG>2  — Crafting a Multi‑Turn Evaluation Dataset</STRONG></H2> <P><SPAN>Each row holds the </SPAN><STRONG>full conversation so far</STRONG><SPAN> plus the grading guidelines for the next response:</SPAN></P> <LI-CODE lang="python">eval_set = [ { "request": { "messages": [ {"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello! How can I help you today?"}, {"role": "user", "content": "Tell me a joke"} ] }, "guidelines": [ "The response should be humorous but appropriate", "The response should be concise" ] }, { "request": { "messages": [ {"role": "user", "content": "What's the weather like?"}, {"role": "assistant", "content": "I don't have real‑time weather data. You'd need to check a weather service for that information."}, {"role": "user", "content": "Can you explain how LLMs work?"} ] }, "guidelines": [ "The response should be technical but accessible", "The response should include a brief explanation of attention mechanisms" ] } ] # Convert to a DataFrame so mlflow.evaluate can treat it as a table‑like object eval_df = pd.DataFrame(eval_set)</LI-CODE> <P><SPAN>I keep it in Pandas because it’s easy to version and quick to inspect.</SPAN></P> <H2><STRONG>3  — Implementing a Simple Agent (Demo Only)</STRONG></H2> <P><SPAN>Here’s a throw‑away rule‑based agent. The only thing that matters is the </SPAN><STRONG>function signature</STRONG><SPAN>—</SPAN><SPAN>messages</SPAN><SPAN> comes in as the full dialog.</SPAN></P> <LI-CODE lang="python">@mlflow.trace(span_type="AGENT") def my_agent(messages): """A trivial rule‑based agent for illustration.""" last_user_message = next((m["content"] for m in reversed(messages) if m["role"] == "user"), "") if "joke" in last_user_message.lower(): return "Why did the AI go to art school? To learn how to draw conclusions!" elif "weather" in last_user_message.lower(): return "I don't have access to real‑time weather data, but I can help you understand weather patterns in general." elif any(term in last_user_message.lower() for term in ["llm", "language model"]): return ( "Large Language Models (LLMs) are AI systems trained on vast amounts of text data. " "They use transformer architectures with attention mechanisms to model relationships between tokens." ) else: return f\"I understand you asked about: '{last_user_message}'. How can I help with that?\"</LI-CODE> <P><STRONG>mlflow.trace</STRONG><SPAN> gives me latency and nested‑call traces for free—handy once I replace this with a real chain‑of‑thought or RAG pipeline.</SPAN></P> <H2><STRONG>4  — Global Guidelines</STRONG></H2> <P><SPAN>I add a second layer of checks that apply to <STRONG>every</STRONG> row:</SPAN></P> <LI-CODE lang="python">global_guidelines = { "helpfulness": ["The response must be helpful and directly address the user's question"], "clarity": ["The response must be clear and well‑structured"], "safety": ["The response must be safe and appropriate"] } </LI-CODE> <H2><STRONG>5  — Running the Evaluation</STRONG></H2> <P><SPAN>Now let’s grade the agent. Everything lives inside an MLflow run so I can track it later:</SPAN></P> <LI-CODE lang="python">with mlflow.start_run(run_name="agent_evaluation_v1") as run: evaluation_results = mlflow.evaluate( data=eval_df, model=lambda request: my_agent(**request), model_type="databricks-agent", evaluator_config={ "databricks-agent": { "global_guidelines": global_guidelines } } )</LI-CODE> <P><SPAN>Under the hood, Databricks calls proprietary expert LLM judges and returns scores like </SPAN><SPAN>helpfulness_score</SPAN><SPAN>, </SPAN><SPAN>clarity_score</SPAN><SPAN>, and </SPAN><SPAN>safety_score</SPAN><SPAN>.</SPAN></P> <H2><STRONG>6  — Inspecting the Results</STRONG></H2> <LI-CODE lang="python">print("Aggregated metrics:\n", evaluation_results.metrics) per_request_results = evaluation_results.tables["eval_results"] print("\nPer‑request results:\n", per_request_results)</LI-CODE> <P><SPAN>Need a quick visual? In a notebook just run:</SPAN></P> <LI-CODE lang="python">display(per_request_results)</LI-CODE> <P><SPAN>Aggregates catch regressions; per‑request rows tell me exactly </SPAN><STRONG>which turn</STRONG><SPAN> broke the guideline.</SPAN></P> <H2><STRONG>7  — Persisting Metrics to Delta</STRONG></H2> <P><SPAN>I push every run into Delta so I can chart trends and set alerts:</SPAN></P> <LI-CODE lang="python">def append_metrics_to_table(run_name, mlflow_metrics, delta_table_name): data = {k: v for k, v in mlflow_metrics.items() if "error_count" not in k} data.update({"run_name": run_name, "timestamp": pd.Timestamp.now()}) (spark.createDataFrame([data]) .write.mode("append") .saveAsTable(delta_table_name)) # append_metrics_to_table("agent_evaluation_v1", evaluation_results.metrics, "catalog.schema.agent_eval_results") </LI-CODE> <P><SPAN>Hook this into DLT or a Databricks SQL dashboard and you’ve got continuous monitoring.</SPAN></P> <H2><STRONG>Wrap‑Up &amp; Next Steps</STRONG></H2> <P><SPAN>You now have a </SPAN><STRONG>repeatable, multi‑turn evaluation loop</STRONG><SPAN> that is:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Reproducible</STRONG><SPAN> – every run is logged in MLflow.</SPAN><SPAN><BR /><BR /></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Swappable</STRONG><SPAN> – replace the toy agent with your production model; the harness stays the same.</SPAN><SPAN><BR /><BR /></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Observable</STRONG><SPAN> – metrics and traces live in Delta and MLflow for long‑term visibility.</SPAN><SPAN><BR /><BR /></SPAN></LI> </UL> <H3><STRONG>Where I go from here</STRONG></H3> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Expand the eval set with adversarial prompts and longer chats.</SPAN><SPAN><BR /><BR /></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Track latency, token usage, and cost alongside quality scores.</SPAN><SPAN><BR /><BR /></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Wire this into CI/CD—block model promotion if helpfulness drops.</SPAN><SPAN><BR /><BR /></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Store everything in Unity Catalog for governance.</SPAN><SPAN><BR /><BR /></SPAN></LI> </UL> <P><SPAN>Embed these checks early and your agents will improve with every release—no surprises when they hit real users. If you have questions or tweaks, ping me anytime. Happy shipping!</SPAN></P> <P>&nbsp;</P> Tue, 08 Apr 2025 14:16:45 GMT Debu-Sinha 2025-04-08T14:16:45Z https://community.databricks.com/t5/technical-blog/evaluate-multi-turn-chatbots-on-databricks-with-mlflow-a-step-by/ba-p/114741 <P>Think your chatbot is ready for prime time? In this hands‑on walkthrough I show you—step by step—how I grade multi‑turn conversations on Databricks, surface clarity‑ and safety‑scores with MLflow, and log everything to Delta so regressions never slip through. Five minutes of setup, lifetime of confidence.</P> Tue, 08 Apr 2025 14:16:45 GMT https://community.databricks.com/t5/technical-blog/evaluate-multi-turn-chatbots-on-databricks-with-mlflow-a-step-by/ba-p/114741 Debu-Sinha 2025-04-08T14:16:45Z