Trigger.AvailableNow getting stuck when there is no event
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-01-2023 09:44 AM
Hi, I have several streaming jobs, however one of them uses the Trigger.AvailableNow. The issue is that it gets stuck when there is no events or finishes ingesting all events. The expected behavior would be the job being shutdown.
I've already checked and there is no new messages in my SQS Queue.
I'm using scala and cluster 12.2 LTS
Driver: Standard_DS3_v2 · Workers: Standard_D8s_v3 · 2-8 workers · 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)
Has anyone seen this before?
- Labels:
-
Autoloader
-
Streaming
-
Trigger.AvailableNow
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-14-2023 02:06 PM
what is your source? what is your trigger value? have you check the driver logs for any WARN or ERROR messages while the job is running?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-14-2023 04:03 PM
Hi, the source is an S3 bucket using file notification with SQS.
No errors or warns in the logs, the AvailableNow trigger just gets stuck.

