Custom Agent traces only showing basic input and output

taniumalloy
New Contributor III

Monday

I've been working with MLflow Agent Server deployments using the OpenAI Agents SDK hosted both locally and in Databricks Apps. Currently, tracing is instrumented with:

set_trace_processors([])
mlflow.openai.autolog()

Locally, everything works great and my traces show annotations for tool calls and include lots of rich telemetry: 

taniumalloy_0-1787599425876.png

However, when I migrate to Databricks apps and try to leveraged Databricks Experiments, I lose all the details. It just shows the basic input and output text:

taniumalloy_1-1787599561475.png

I've tested with both LangChain and OpenAI Agents SDK apps and nothing has changed on the Databricks side.

Is there currently limited support for tracing custom agents? Or has anyone been able to get proper tracing going with custom agents? 

Following the guide here: https://github.com/databricks/app-templates/blob/main/agent-openai-agents-sdk/agent_server/agent.py

 

0 Kudos

taniumalloy
New Contributor III

Monday

After some more testing, it looks like you need to store the traces in Unity Catalog to get the details I was expecting. When using the "In the Experiment" option, functionality is limited. 

taniumalloy_0-1787623922515.png

 

0 Kudos

balajij8
Esteemed Contributor II

Tuesday

@taniumalloy Unity Catalog trace storage provides additional benefits beyond trace detail - unlimited storage, queryability for analysis and debugging, Unity Catalog governance and permissions and compatibility with the latest features. Trace richness is significant in it. You can always choose Unity Catalog as the trace storage location for agent deployment.

View solution in original post