Hi there, How do I use the azure databricks dlt pipeline to consume azure Event Hub data
Code :
TOPIC = "myeventhub"
KAFKA_BROKER = ""
GROUP_ID = "group_dev"
raw_kafka_events = (spark.readStream .format("kafka")
.option("subscribe", EH_NAME)
.option("kafka.bootstrap.servers", KAFKA_BROKER)
.option("kafka.group.id", GROUP_ID) # Set Kafka consumer group ID .option("kafka.security.protocol", "SASL_SSL")
.option("kafka.sasl.jaas.config", f"kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$ConnectionString\" password=\"{EH_CONN_STR}\";") # .option("kafka.ssl.endpoint.identification.algorithm", "https") # .option("kafka.sasl.mechanism", "PLAIN")
.option("failOnDataLoss", "false")
.option("startingOffsets", "earliest")
.load()
)
parsed_data = raw_kafka_events.select(col("value").cast("string")).alias("event") display(parsed_data)
Error log:
kafkashaded.org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
Caused by: kafkashaded.org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config at kafkashaded.org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184) at kafkashaded.org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192) at kafkashaded.org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) at kafkashaded.org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) at kafkashaded.org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:522)
Thanks&Regards, zmsoft
