Can I connect to Eventhub using Kafka API? Which is the preferred way?

[brickster_2018 · Accepted solution]

Yes, it's possible to use Kafka API to connect to the eventhub. Eventhub supports the usage of Kafka API to stream the data from the Eventhub

Reference:

https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview

Sample program:

%python
 
eh_sasl = 'kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://myazure-train-eventhub.us.servicebus.windows.net/;SharedAccessKeyName=manage-policy;SharedAccessKey=xxxxxXXXXXXXXX/xxxxxxx;';
 
inputStream = spark.readStream \
.format("kafka") \
.option("subscribe", "dataabricks-stream-event") \
.option("kafka.bootstrap.servers", "azure-training-us.servicebus.windows.net:9093") \
.option("kafka.sasl.mechanism", "PLAIN") \
.option("startingOffsets","earliest") \
.option("kafka.security.protocol", "SASL_SSL") \
.option("kafka.sasl.jaas.config", eh_sasl) \
.option("kafka.request.timeout.ms", "60000") \
.option("kafka.session.timeout.ms", "30000") \
.option("failOnDataLoss", "false") \
.load()