Hello team,
We are researching the streaming capabilities of our data platform and currently in need of reading data from EVH ( event hub) with our Databricks notebooks. Unfortunately there seems to be an error somewhere due to Timeout Exception: Timed out waiting for a node assignment. Call: describe Topics.
Can you please help us tackle it?
KAFKA_OPTIONS = {
"kafka.bootstrap.servers" : f"{EH_NAMESPACE}.servicebus.windows.net:9093",
"subscribe" : EH_NAME,
"kafka.sasl.mechanism" : "PLAIN",
"kafka.security.protocol" : "SASL_SSL",
"kafka.group.id" : "databricks-streaming-eh-kelvin",
"kafka.sasl.jaas.config" : f"kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$ConnectionString\" password=\"{EH_CONN_STR}\";",
"kafka.request.timeout.ms" : 60000, # https://learn.microsoft.com/en-us/azure/event-hubs/apache-kafka-configurations#consumer-configurations-only
"kafka.session.timeout.ms" : 300000,
"maxOffsetsPerTrigger" : 4000,
"failOnDataLoss" : True,
"startingOffsets" : "latest"
}
df = (spark.readStream
.format("kafka")
.options(**KAFKA_OPTIONS)
.load())
Generates the proper schema, so some sort of handshake happened between EVH and DBR
However if I try to display my def.:
display(df)
java.util.concurrent.ExecutionException: kafkashaded.org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeTopics <p><strong>Caused by: kafkashaded.org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeTopics</strong></p><p> </p>