Trigger.AvailableNow getting stuck when there is no event

[fhmessas]

Hi, I have several streaming jobs, however one of them uses the Trigger.AvailableNow. The issue is that it gets stuck when there is no events or finishes ingesting all events. The expected behavior would be the job being shutdown.

I've already checked and there is no new messages in my SQS Queue.

I'm using scala and cluster 12.2 LTS

Driver: Standard_DS3_v2 · Workers: Standard_D8s_v3 · 2-8 workers · 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)

 

Has anyone seen this before?

[jose_gonzalez]

what is your source? what is your trigger value? have you check the driver logs for any WARN or ERROR messages while the job is running?

[fhmessas]

Hi, the source is an S3 bucket using file notification with SQS.

No errors or warns in the logs, the AvailableNow trigger just gets stuck.