trying to send data from a stream table to an azure event hub in a serverless cluster

[Areqio]

Is there a way to stream data from Databricks to Azure event hubs in a serverless pipeline environment without using the azure-eventhub library, since it isn’t compatible with serverless pipelines, and instead rely solely on the Kafka-compatible interface?

[amirabedhiafi]

Hello @Areqio !

Yes, you can use Azure event hubs through its Kafka compatible endpoint and not the azure-eventhubs-spark / azure-eventhub connector. JVM libraries are not allowed in LSDP and event hubs should be accessed through the built in Spark Kafka connector.

For writing from a streaming table to event hubs, you use the pipeline sink API with format = "kafka" and an append_flow.

The event hub name is used as the Kafka topic and the namespace is the Kafka bootstrap server on port 9093. Event hubs supports this Kafka endpoint in standard, premium and dedicated tiers not basic. 

For the Kafka writer, the outgoing dataframe must contain a value column and it can optionally contain key, topic, partition and headers.

The only limitation is that the sink API is python only and supports streaming queries only and must be used through append_flow and a full refresh does not clean up external sink data so replayed data can be appended again.

[Areqio]

Thank you for the reply! do you have a code example? I was able to do it for a delta table through a notebook in a serverless cluster using kafka, but cannot do it for a streaming table nor in a dlt pipeline