Delta Live Table Pipeline to EventHub

[NIK251]

i want to read and load the data to eventhub. And there is an error message:org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 138.0 failed 4 times, most recent failure: Lost task 0.3 in stage 138.0 (TID 177) (10.139.64.4 executor driver): java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.OperationCancelledException: Entity(anomalydetection-eventhub): send failed while dispatching to Reactor, see cause for more details.

Below is my code 

from pyspark.sql.functions import to_json, struct

silver_data = spark.readStream.format("delta").table("anomaydetection_processed.silver_clean_data")

raw_connection_string = dbutils.secrets.get(scope="anomalydetect-scope", key="connection-string-eventhub-send")

ehConf = {}

ehConf['eventhubs.connectionString'] = spark._sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(raw_connection_string)

output_df = silver_data.select(to_json(struct("*")).alias("body"))

output_df.writeStream \

    .format("eventhubs") \

    .options(**ehConf) \

    .option("checkpointLocation", "/mnt/anomalydetection2/presentation/eventhub_checkpoint") \

    .start()

[Sidhant07]

Hi,

The error originates from the eventhub connector. Kindly check this with the EventHub Spark connector team.

Please use the latest connector. https://github.com/Azure/azure-event-hubs-spark

There are known issues with the Event hub connector like below.
https://github.com/Azure/azure-event-hubs-spark/issues/612#issuecomment-958098093

[NIK251]

Thanks for your response, i have checked that my cluster use the latest connector 2.3.22 and use the 15.4 LTS version. Are there any changes my error come from my cluster is in singlenode?

[Sidhant07]

can you try with shared/multinode cluster as well to see if that resolves the issue.

I looked for similar tickets from other customers and most likely the issue is from eventhub.

[NIK251]

sorry for the inconvenience that i cant try with multi node cluster now. Are there another ways? @@