Can I connect to Eventhub using Kafka API? Which is the preferred way?

brickster_2018
Databricks Employee
Databricks Employee
 

brickster_2018
Databricks Employee
Databricks Employee

Yes, it's possible to use Kafka API to connect to the eventhub. Eventhub supports the usage of Kafka API to stream the data from the Eventhub

Reference:

https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview

Sample program:

%python
 
eh_sasl = 'kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://myazure-train-eventhub.us.servicebus.windows.net/;SharedAccessKeyName=manage-policy;SharedAccessKey=xxxxxXXXXXXXXX/xxxxxxx;';
 
inputStream = spark.readStream \
.format("kafka") \
.option("subscribe", "dataabricks-stream-event") \
.option("kafka.bootstrap.servers", "azure-training-us.servicebus.windows.net:9093") \
.option("kafka.sasl.mechanism", "PLAIN") \
.option("startingOffsets","earliest") \
.option("kafka.security.protocol", "SASL_SSL") \
.option("kafka.sasl.jaas.config", eh_sasl) \
.option("kafka.request.timeout.ms", "60000") \
.option("kafka.session.timeout.ms", "30000") \
.option("failOnDataLoss", "false") \
.load()

View solution in original post