Every system you run generates a constant stream of signals: traces that show how a request travelled through your service, logs that capture what happened and why, and metrics that measure the overall health. This set of telemetry is among the most valuable operational data your organization produces, yet for most teams it never makes it to the place where they do their best analytical work: their lakehouse.
Today, we're announcing Zerobus Ingest now natively supports the OpenTelemetry Protocol (OTLP), allowing you to bring OpenTelemetry (OTEL) data directly into your lakehouse. Zerobus Ingest OTEL is now available in Beta, which allows you to stream traces, logs, and metrics directly into Unity Catalog Delta tables, using standard OpenTelemetry SDKs and collectors you likely already have in place, with no custom libraries or intermediary pipelines required.
What is OpenTelemetry?
OpenTelemetry is an open source standard for instrumenting applications and collecting telemetry data. It is also vendor and tool agnostic, able to be used with a broad variety of tools. OpenTelemetry defines a common framework and toolkit to facilitate the generation, export, and collection of three types of signals:
- Traces: The story of a single request as it moves through your system. Each trace consists of spans, which are individual operations with timing, status, and attributes. Together, they let you reconstruct exactly what happened during a user interaction or job run.
- Logs: Timestamped records of events. OpenTelemetry logs can be correlated with traces via trace_id and span_id, connecting discrete events to the broader request context.
- Metrics: Aggregated numerical measurements over time: counters, gauges, histograms, and more. Metrics answer questions like "what is my error rate right now?" or "how is CPU utilization trending?"
OpenTelemetry has become the dominant open source framework for observability. The ecosystem of supported components and integrations is broad, with SDKs for every major language, and a large number of tools, agents, and collectors which already speak the OTLP protocol out-of-the-box.
The standard telemetry pipeline
When you instrument an application with OpenTelemetry, the data typically flows through three layers:
- The application emits raw telemetry.
- A collector or agent — such as the OpenTelemetry Collector, Fluent Bit, Telegraf, or syslog-ng — receives, processes, and forwards that telemetry to a backend.
- The backend is where the data lands and gets stored.
Until now, that backend was almost always a proprietary observability vendor. Zerobus Ingest OTEL is the backend, the endpoint your collectors point to, which will publish data directly in your Delta tables.
The problem with existing approaches
Before this integration, getting telemetry data into Databricks required extra steps. The most common patterns were:
- Extracting data from a third-party observability platform: Many teams initially send telemetry to a proprietary vendor, then struggle to retrieve it. Retrieval methods vary (e.g., vendor publishing to a Kafka topic, or polling export APIs), but all depend on vendor support, are subject to vendor retention limits, and often incur egress costs to access one's own data.
- Dual-writing to object storage: Alternatively, configure instrumentation to write telemetry files (JSON, Parquet, logs) directly to object storage (S3/ADLS), then ingest into Delta using Auto Loader or batch jobs. While offering more control, this requires two separate outputs per system and a continuous pipeline to monitor and load new files.
- Kafka as an intermediary bus: Routing all telemetry via a central Kafka broker and then consuming it into Delta works well at scale but incurs significant infrastructure overhead. This includes managing the Kafka cluster, monitoring consumer lag, handling rebalancing, and maintaining the consumer code—all just to move data.
All three patterns work, but they share the same underlying problem: operational complexity that has nothing to do with the telemetry itself. You're managing brokers, pipelines, export configurations, and ingestion jobs just to get data into a place where you can finally do something with it.
Zerobus Ingest removes that middle layer entirely. Your collector points directly at the Zerobus Ingest OTEL endpoint, and the data lands in your Delta tables — no broker, no export job, no polling pipeline, no extra monitoring surface.
Why land telemetry in the lakehouse?
This might seem like a natural question: observability vendors already provide dashboards, alerting, and querying over this data. So why bring it into the lakehouse?
You own the data
When telemetry lives in a proprietary observability platform, it lives there on their terms. You're accessing your own data through their interface, subject to their retention policies, their pricing model, and their query language. When that data lands in Unity Catalog, it's in your managed storage. You control it. You can query it with whatever tools you want, keep it as long as you want, and move it if you need to.
It's dramatically cheaper for long-term storage
Proprietary observability vendors typically store data in high-performance indexed systems optimized for real-time dashboards. That performance comes at a cost. For real-time alerting, that tradeoff makes sense. But for data you want to keep for 6 months or a year — historical analysis, capacity planning, compliance — Delta on object storage is orders of magnitude cheaper. You're paying for bytes in object storage, not seats in a SaaS platform.
You can do an analysis that proprietary tools can't
Once telemetry is in Delta, it becomes just another table in your lakehouse. That means:
- Join with business data: Correlate API trace latency with revenue impact. Join error logs with customer records. Connect infrastructure metrics to product usage patterns.
- Historical trend analysis: Query what your system looked like a year ago. Build dashboards showing how load has changed over time. Do capacity forecasting using actual historical data, not a 30-day rolling window.
- SQL with Databricks SQL: Your data engineers and analysts already know SQL. They can query your telemetry just as they query everything else.
- AI and ML: Train anomaly detection models on your metrics history. Build evaluations for AI agents using execution traces logged to Delta. Audit what your AI systems are doing and why over time. As AI becomes more central to how products are built, the ability to log, retain, and analyze agent behaviour at scale becomes increasingly important and the lakehouse is the right place for that.
- Security and observability: Telemetry is security data too. Traces and logs capture what your systems are doing and who is doing it. Streaming this data into the lakehouse enables security analysis, anomaly detection, and long-term audit logs — all using the same data platform your data team already works in.
What tools work with Zerobus Ingest OTEL?
Any OTLP-compatible exporter or collector can send data to Zerobus Ingest. Some common examples:
- Fluent Bit — A lightweight log processor and forwarder widely used in Kubernetes environments. It supports OTLP output natively.
- Telegraf — A plugin-driven metrics agent that can collect from hundreds of sources and forward via OTLP output.
- syslog-ng — A flexible log management tool with OTLP output support, widely used for centralizing logs from diverse infrastructure. Read the companion blog on how to setup streaming of syslog-ng data to your lakehouse, powered by Zerobus Ingest OTEL.
If your tool speaks OTLP/gRPC, it can send data to Zerobus Ingest.
Getting started (Beta)
This feature is in Beta. Here's what you need before you can start sending data:
Prerequisites
- A Zerobus Ingest endpoint — Your endpoint URL is derived from your Databricks workspace ID and region (e.g., 1234567890123456.zerobus.us-west-2.cloud.databricks.com).
- Target Delta tables — Tables must be created in advance with the required schemas. Zerobus does not auto-create tables. Each signal type (traces, logs, metrics) needs its own table. Full CREATE TABLE statements are in the table reference docs.
- A service principal with OAuth credentials — Grant the service principal access to your target tables. See Create a service principal and grant permissions in the Zerobus Ingest docs.
Configure your exporter
Point your OTLP exporter at the Zerobus endpoint and include two required headers on every request:
- x-databricks-zerobus-table-name — The fully-qualified Unity Catalog table name (catalog.schema.table).
- Authorization — An OAuth bearer token from your service principal.
Each signal type is a separate gRPC service path, so you'll configure separate exporters for traces, logs, and metrics — each pointing to its own table.
Note: Currently, only OTLP/gRPC (Protobuf) is supported. HTTP/Protobuf support is on the roadmap.
For complete setup instructions, including Python SDK examples and OpenTelemetry Collector YAML configuration, see the full documentation.
Unlocking your telemetry data
The ability to stream OpenTelemetry data directly into your lakehouse is a small configuration change with a large surface area of impact. Your telemetry is no longer trapped in a proprietary system with a 30-day retention window or locked behind a per-seat bill. It lives in Delta, alongside your business data, queryable by your data teams, trainable by your ML models, and retainable for as long as you need it.
This is now available in Beta, and we're actively developing this feature. If you try it out, we'd love to hear what you build and where you want us to take this project!
Get started: Zerobus Ingest OpenTelemetry documentation
Example: syslog-ng-zerobus
Have questions or feedback? Drop them in the comments below.
