Hello team, 

We are researching the streaming capabilities of our data platform and currently in need of reading data from EVH ( event hub) with our Databricks notebooks. Unfortunately there seems to be an error somewhere due to Timeout Exception: Timed out waiting for a node assignment. Call: describe Topics.

Can you please help us tackle it? 

• The connections were assembled as per article https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/event-hubs ;
• EVH networking is set to Public access 
• DBR runtime is set as "spark version": "15.4.x-scala2.12"

KAFKA_OPTIONS = {
  "kafka.bootstrap.servers"  : f"{EH_NAMESPACE}.servicebus.windows.net:9093",
  "subscribe"                : EH_NAME,
  "kafka.sasl.mechanism"     : "PLAIN",
  "kafka.security.protocol"  : "SASL_SSL",
  "kafka.group.id"           : "databricks-streaming-eh-kelvin",
  "kafka.sasl.jaas.config"   : f"kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$ConnectionString\" password=\"{EH_CONN_STR}\";",
  "kafka.request.timeout.ms" : 60000, # https://learn.microsoft.com/en-us/azure/event-hubs/apache-kafka-configurations#consumer-configurations-only
  "kafka.session.timeout.ms" : 300000,
  "maxOffsetsPerTrigger"     : 4000,
  "failOnDataLoss"           : True,
  "startingOffsets"          : "latest"
}

df = (spark.readStream
    .format("kafka")
    .options(**KAFKA_OPTIONS)
    .load())

display(df)
java.util.concurrent.ExecutionException: kafkashaded.org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeTopics ​<p><strong>Caused by: kafkashaded.org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeTopics</strong></p><p> </p>