topic Re: messages from event hub does not flow after a time in Data Engineering

messages from event hub does not flow after a time

Jreco — Tue, 01 Feb 2022 11:01:03 GMT

Hi Team,

I'm trying to build a Real-time solution using Databricks and Event hubs.

Something weird happens after a time that the process start.

At the begining the messages flow through the process as expected with this rate:

please, note that the last updated time is 50Sec.

However, after a time, the messages don't flow:

Please note that the last updated time is: 11 hours.

If I restart the job, the messages flow again as expected (even recovering the messages that does not were processed in the las 11 hours, for this case)

This is a graph of the example of the issue:

The last peek was when I restarted the job.

Any idea what could happens?

Re: messages from event hub does not flow after a time

Hubert-Dudek — Tue, 01 Feb 2022 15:01:12 GMT

please check is .option("checkpointLocation", "/mnt/your_storage/") specified for structured streaming,
it can also depend what is than done with stream (writeStream),
connection to eventhub is quite straight forward, please verify also flow on Azure side as there we can see streaming messages in real time (go to entities, select Even Hub, in "Futures" select "process data" than "Explore" than "Create")

Re: messages from event hub does not flow after a time

Jreco — Tue, 01 Feb 2022 15:18:38 GMT

Thanks for your answer @Hubert Dudek ,

Is already specified
What do youn mean with this?
This is the weird part of this, bucause the data is flowing good, but at any time is like the Job stop the reading or somethign like that and if I restart the job, all continues working well

Re: messages from event hub does not flow after a time

Hubert-Dudek — Tue, 01 Feb 2022 18:38:36 GMT

I mean that you read stream in some purpose, usually to transform it and write it somewhere. So problem can be not with reading but writing part.

Re: messages from event hub does not flow after a time

Jreco — Tue, 01 Feb 2022 19:52:12 GMT

I'm assuming that the issue is not the Writing part because the DB does not present any kind of blockers or conflicts.

Re: messages from event hub does not flow after a time

jose_gonzalez — Wed, 09 Feb 2022 01:14:29 GMT

hi @Jhonatan Reyes ,

Do you control/limit the max number of events process per trigger in your event hubs? check "maxEventsPerTrigger" or Whats your trigger internal? also, how many partitions are you reading from? whats your sink?

Re: messages from event hub does not flow after a time

jose_gonzalez — Wed, 16 Feb 2022 19:49:05 GMT

@Jhonatan Reyes

Do you still need help with this or the issue has been mitigated/solved?