cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Generative AI
Explore discussions on generative artificial intelligence techniques and applications within the Databricks Community. Share ideas, challenges, and breakthroughs in this cutting-edge field.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Monitoring a custom Agent (Responses) serving endpoint with an experiment on the Unity Catalog

FantineM
New Contributor III

Hi,

I have a Langgraph agent deployed in Databricks on a custom serving endpoint of type Agent (Responses).
So far, I have been tracing this agent on a "classic" monitoring experiment. I was using an ad hoc job to sync those traces to a table in the Unity Catalog. But now, one can create an experiment with a location in the UC directly, which sounds handy for my purpose. However, after I redeploy the agent (I set the mlflow experiment to the new one with UC location before deploying and I set 
MLFLOW_EXPERIMENT_ID as en environment variable for the deployed entity) I cannot see any traces in the new experiment.
The only way I could ever make this work was creating a span from a notebook and running a non deployed agent from there.
Am I missing something? Has anyone succeeded to make it work who could help me out?
Thanks in advance for any help or suggestion ๐Ÿ˜‰

1 ACCEPTED SOLUTION

Accepted Solutions

tamara48wells
New Contributor III

When a deployed agent serving endpoint fails to log traces to a Unity Catalog-backed experiment, it is typically due to permissions or environment configuration issues. First, ensure that the service principal running the serving endpoint has explicit USE_CATALOG, USE_SCHEMA, and write permissions on the target Unity Catalog schema. Second, verify that the environment variables passed to the endpoint include MLFLOW_TRACKING_URI=databricks alongside the correct experiment path/name configuration (using MLFLOW_EXPERIMENT_NAME rather than just MLFLOW_EXPERIMENT_ID), as model serving containers require proper tracking URI context to authenticate and route OpenTelemetry/MLflow traces to Unity Catalog.

View solution in original post

2 REPLIES 2

tamara48wells
New Contributor III

When a deployed agent serving endpoint fails to log traces to a Unity Catalog-backed experiment, it is typically due to permissions or environment configuration issues. First, ensure that the service principal running the serving endpoint has explicit USE_CATALOG, USE_SCHEMA, and write permissions on the target Unity Catalog schema. Second, verify that the environment variables passed to the endpoint include MLFLOW_TRACKING_URI=databricks alongside the correct experiment path/name configuration (using MLFLOW_EXPERIMENT_NAME rather than just MLFLOW_EXPERIMENT_ID), as model serving containers require proper tracking URI context to authenticate and route OpenTelemetry/MLflow traces to Unity Catalog.

FantineM
New Contributor III

Hi! Thank you so much for your reply!
So in the end, just adding MLFLOW_TRACKING_URI as env variables to the entity did the trick. Permissions were already set properly.