Structured Streaming writeStream - Query is no longer active causes task to fail
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-19-2025 03:26 AM
Hi,
I execute readStream/writeStream in workflow task.
Write stream uses .trigger(availableNow=True) option. After writeStream I'm waiting query to finish with
"Query <id> is no longer active" error which causes task in workflow to fail.
I have also check option:
but the behavior is the same.
Does anybody knows why this error occurs? How to avoid it ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-19-2025 04:44 AM
Hello @p_romm - Are you using serverless compute?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-24-2025 01:00 AM
@Alberto_Umana yes, it is used on serveless, I didn't test it in general compute yet.
This is long running task in foreachBatch function, and in the messages I get also:
Query <id> has not received any updates in 60 seconds. Spark Session ID is <id>
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-24-2025 02:34 AM
However it looks like that 'Query <id> has not received any updates in 60 seconds' is not correlated with "Query <id> is no longer active"
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-28-2025 09:15 AM
@Alberto_Umana this bug was apparently fixed a few months ago, but we're still facing the same issue on our end.
