topic Re: Trigger.AvailableNow getting stuck when there is no event in Data Engineering

Trigger.AvailableNow getting stuck when there is no event

fhmessas — Thu, 01 Jun 2023 16:44:48 GMT

Hi, I have several streaming jobs, however one of them uses the Trigger.AvailableNow. The issue is that it gets stuck when there is no events or finishes ingesting all events. The expected behavior would be the job being shutdown.

I've already checked and there is no new messages in my SQS Queue.

I'm using scala and cluster 12.2 LTS

Driver: Standard_DS3_v2 · Workers: Standard_D8s_v3 · 2-8 workers · 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)

Has anyone seen this before?

Re: Trigger.AvailableNow getting stuck when there is no event

jose_gonzalez — Wed, 14 Jun 2023 21:06:30 GMT

what is your source? what is your trigger value? have you check the driver logs for any WARN or ERROR messages while the job is running?

Re: Trigger.AvailableNow getting stuck when there is no event

fhmessas — Wed, 14 Jun 2023 23:03:03 GMT

Hi, the source is an S3 bucket using file notification with SQS.

No errors or warns in the logs, the AvailableNow trigger just gets stuck.