topic Re: How to improve Spark Streaming writer Input Rate and Processing rate? in Data Engineering

How to improve Spark Streaming writer Input Rate and Processing rate?

RengarLee — Fri, 29 Apr 2022 04:13:56 GMT

Hi!

I have many questions about Spark Streaming and Evnethub。

Can you help me?

Q1:How to improve Spark Streaming writer Input Rate and Processing rate?

I connect Azure Eventhubs using Spark Streaming(Azure Databricks), but I found if I use display, this input rate is very quick, if I use writer is very Slow. the result this Prcture.1, the code this Picture.2 and Picture.3. I want to improve the writer input rate and processing rate to the extent that the outgoing bytes are Greater than the Incoming bytes in the event hub, like the display.

What should I do?

Q2: setMaxEventsPerTrigger not equal to numInputRow?

I set 10000 to setMaxEventsPerTrigger on eventhubsConf, but why numInputRow inside RawData is 1000, like the Prcture.5.setMaxEventsPerTrigger not equal to numInputRow?

Re: How to improve Spark Streaming writer Input Rate and Processing rate?

jose_gonzalez — Fri, 29 Apr 2022 21:12:49 GMT

Hi @Rengar Lee ,

How many Eventhubs partitions are you reading from? check your Ganglia UI to check your cluster utilization. also, whats the time it takes to write the data to the sink? you can get the query metrics from the Spark logs.

Re: How to improve Spark Streaming writer Input Rate and Processing rate?

RengarLee — Sat, 30 Apr 2022 00:06:25 GMT

How many Eventhubs partitions are you reading from?

Only 1 partition.

Check your Ganglia UI to check your cluster utilization

Prcture.1

What's the time it takes to write the data to the sink?

Prcture.2

I found that no matter how much setMaxEventsPerTrigger the numInputRow is 1000.

Can't finish writing in 1 second， which causes a delay for the next write， so event hubs' outgoing bytes are Low.

if I can set numInputRow to exceed 1000， I think the question can resolve.

EventHubs is in Prcture3.

Re: How to improve Spark Streaming writer Input Rate and Processing rate?

RengarLee — Sat, 30 Apr 2022 00:06:37 GMT

Re: How to improve Spark Streaming writer Input Rate and Processing rate?

RengarLee — Sat, 30 Apr 2022 00:06:51 GMT

Re: How to improve Spark Streaming writer Input Rate and Processing rate?

RengarLee — Wed, 04 May 2022 07:51:44 GMT

setMaxEventsPerTrigger not equal to numInputRow is my problem