Azure Event Hub throws Timeout Exceptio: Timed out waiting for a node assignment. Call: describeTopi
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Friday
Hello team,
We are researching the streaming capabilities of our data platform and currently in need of reading data from EVH ( event hub) with our Databricks notebooks. Unfortunately there seems to be an error somewhere due to Timeout Exception: Timed out waiting for a node assignment. Call: describe Topics.
Can you please help us tackle it?
- The connections were assembled as per article https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/event-hubs ;
- EVH networking is set to Public access
- DBR runtime is set as "spark version": "15.4.x-scala2.12"
KAFKA_OPTIONS = {
"kafka.bootstrap.servers" : f"{EH_NAMESPACE}.servicebus.windows.net:9093",
"subscribe" : EH_NAME,
"kafka.sasl.mechanism" : "PLAIN",
"kafka.security.protocol" : "SASL_SSL",
"kafka.group.id" : "databricks-streaming-eh-kelvin",
"kafka.sasl.jaas.config" : f"kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$ConnectionString\" password=\"{EH_CONN_STR}\";",
"kafka.request.timeout.ms" : 60000, # https://learn.microsoft.com/en-us/azure/event-hubs/apache-kafka-configurations#consumer-configurations-only
"kafka.session.timeout.ms" : 300000,
"maxOffsetsPerTrigger" : 4000,
"failOnDataLoss" : True,
"startingOffsets" : "latest"
}
df = (spark.readStream
.format("kafka")
.options(**KAFKA_OPTIONS)
.load())
However if I try to display my def.:
display(df)
java.util.concurrent.ExecutionException: kafkashaded.org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeTopics <p><strong>Caused by: kafkashaded.org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeTopics</strong></p><p> </p>
- Labels:
-
Spark
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
yesterday - last edited yesterday
Hi @Ivan_Pyrog ,
java.util.concurrent.ExecutionException: kafkashaded.org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeTopics <p><strong>Caused by: kafkashaded.org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeTopics</strong></p><p> </p>
Looking at the above error it is seems that the above errors are occuring due to no network connectivity between Databricks and Kafka cluster. Please make sure that telnet or nc commands should be successful before trying to execute any code to Kafka cluster.
for example
%sh telnet b-2.scram.cxdevorderbrokerstr<>xyz 14002
Also, make sure to enable SSL connections if Kafka cluster requires SSL connection. Please refer below link for SSL connection code snippet:
[+] https://docs.databricks.com/en/structured-streaming/kafka.html