How do I use the azure databricks dlt pipeline to consume azure Event Center data

[zmsoft]

Hi there, How do I use the azure databricks dlt pipeline to consume azure Event Hub data

Code :

TOPIC = "myeventhub" 
KAFKA_BROKER = "" 
GROUP_ID = "group_dev" 

raw_kafka_events = (spark.readStream .format("kafka") 
.option("subscribe", EH_NAME) 
.option("kafka.bootstrap.servers", KAFKA_BROKER) 
.option("kafka.group.id", GROUP_ID) # Set Kafka consumer group ID .option("kafka.security.protocol", "SASL_SSL") 
.option("kafka.sasl.jaas.config", f"kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$ConnectionString\" password=\"{EH_CONN_STR}\";") # .option("kafka.ssl.endpoint.identification.algorithm", "https") # .option("kafka.sasl.mechanism", "PLAIN") 
.option("failOnDataLoss", "false") 
.option("startingOffsets", "earliest") 
.load() 
)


parsed_data = raw_kafka_events.select(col("value").cast("string")).alias("event") display(parsed_data) 

Error log: 

kafkashaded.org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient

Caused by: kafkashaded.org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config at kafkashaded.org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184) at kafkashaded.org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192) at kafkashaded.org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) at kafkashaded.org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) at kafkashaded.org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:522)

Thanks&Regards, zmsoft

[ashraf1395 · Accepted solution]

Hi there @zmsoft ,

Did you have a look at this ref doc : https://docs.databricks.com/aws/en/dlt/event-hubs
This might help