topic Delta Live Table Pipeline to EventHub in Get Started Discussions

Delta Live Table Pipeline to EventHub

NIK251 — Mon, 09 Dec 2024 07:11:04 GMT

i want to read and load the data to eventhub. And there is an error message:org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 138.0 failed 4 times, most recent failure: Lost task 0.3 in stage 138.0 (TID 177) (10.139.64.4 executor driver): java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.OperationCancelledException: Entity(anomalydetection-eventhub): send failed while dispatching to Reactor, see cause for more details.

Below is my code

from pyspark.sql.functions import to_json, struct

silver_data = spark.readStream.format("delta").table("anomaydetection_processed.silver_clean_data")

raw_connection_string = dbutils.secrets.get(scope="anomalydetect-scope", key="connection-string-eventhub-send")

ehConf = {}

ehConf['eventhubs.connectionString'] = spark._sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(raw_connection_string)

output_df = silver_data.select(to_json(struct("*")).alias("body"))

output_df.writeStream \

.format("eventhubs") \

.options(**ehConf) \

.option("checkpointLocation", "/mnt/anomalydetection2/presentation/eventhub_checkpoint") \

.start()

Re: Delta Live Table Pipeline to EventHub

Sidhant07 — Mon, 09 Dec 2024 09:01:59 GMT

Hi,

The error originates from the eventhub connector. Kindly check this with the EventHub Spark connector team.

Please use the latest connector. https://github.com/Azure/azure-event-hubs-spark

There are known issues with the Event hub connector like below.
https://github.com/Azure/azure-event-hubs-spark/issues/612#issuecomment-958098093

Re: Delta Live Table Pipeline to EventHub

NIK251 — Mon, 09 Dec 2024 12:58:27 GMT

Thanks for your response, i have checked that my cluster use the latest connector 2.3.22 and use the 15.4 LTS version. Are there any changes my error come from my cluster is in singlenode?

Re: Delta Live Table Pipeline to EventHub

Sidhant07 — Mon, 09 Dec 2024 13:08:43 GMT

can you try with shared/multinode cluster as well to see if that resolves the issue.

I looked for similar tickets from other customers and most likely the issue is from eventhub.

Re: Delta Live Table Pipeline to EventHub

NIK251 — Tue, 10 Dec 2024 07:01:15 GMT

sorry for the inconvenience that i cant try with multi node cluster now. Are there another ways? @@