Shipping Unity AI Gateway OTel to other collectors

[Rjdudley]

We've enabled Open Telemetry logs in our Unity AI Gateway and have them shipped to Unity Catalog tables, but I am curious if we can configure the gateway to ship OTel to any other collector?  I could swear at one time I saw documentation for that but I can't find it now.

[balajij8]

Unity AI Gateway currently only supports logging to Unity Catalog Delta tables.

You can read the data that lands in Unity Catalog tables and follow below

1. Query the inference table periodically and forward the data to the external collector
2. Set up a streaming pipeline that reads from the Delta table and pushes to the OTel collector
3. Plan a scheduled export to send the data to the observability platform or use Delta Sharing

You can suggest it as a addition of feature on Databricks Ideas portal

[yogeshsingh · Accepted solution]

Thanks for raising this. Based on the current public documentation, Unity AI Gateway’s documented OpenTelemetry path for coding-agent telemetry is to Databricks-managed Unity Catalog Delta tables, rather than to an arbitrary external OTel collector.

The setup shown in the docs uses Databricks OTLP endpoints such as /api/2.0/otel/v1/metrics and /api/2.0/otel/v1/logs, with the Unity Catalog destination table specified through the X-Databricks-UC-Table-Name header.

You may be remembering MLflow tracing, which is documented separately and does support exporting to an OpenTelemetry Collector. That documentation explicitly describes both “OpenTelemetry only” and “Dual export” modes.

Relevant docs:

• Unity AI Gateway coding-agent OTel setup: Integrate with coding agents

• MLflow tracing OTel export: OpenTelemetry Export

So the short answer is: not as a documented Unity AI Gateway capability today, but Databricks does document external OTel collector export for MLflow tracing.

[Rjdudley]

@yogeshsingh Thank you  that's exactly what I had confused in my brain.  @balajij8  as you suggested I will add a feature request.  It probably wouldn't be difficult to roll our own log shipping but endpoint config would be smoother and "platform level".