02-01-2022 03:01 AM
Hi Team,
I'm trying to build a Real-time solution using Databricks and Event hubs.
Something weird happens after a time that the process start.
At the begining the messages flow through the process as expected with this rate:
please, note that the last updated time is 50Sec.
However, after a time, the messages don't flow:
Please note that the last updated time is: 11 hours.
If I restart the job, the messages flow again as expected (even recovering the messages that does not were processed in the las 11 hours, for this case)
This is a graph of the example of the issue:
The last peek was when I restarted the job.
Any idea what could happens?
02-01-2022 07:01 AM
02-01-2022 07:01 AM
02-01-2022 07:18 AM
Thanks for your answer @Hubert Dudek ,
02-01-2022 10:38 AM
I mean that you read stream in some purpose, usually to transform it and write it somewhere. So problem can be not with reading but writing part.
02-01-2022 11:52 AM
I'm assuming that the issue is not the Writing part because the DB does not present any kind of blockers or conflicts.
02-08-2022 05:14 PM
hi @Jhonatan Reyes ,
Do you control/limit the max number of events process per trigger in your event hubs? check "maxEventsPerTrigger" or Whats your trigger internal? also, how many partitions are you reading from? whats your sink?
02-16-2022 11:49 AM
@Jhonatan Reyes
Do you still need help with this or the issue has been mitigated/solved?
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group