Structured Streaming writeStream - Query is no longer active causes task to fail
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-19-2025 03:26 AM
Hi,
I execute readStream/writeStream in workflow task.
Write stream uses .trigger(availableNow=True) option. After writeStream I'm waiting query to finish with
query.awaitTermination(). However from time to time, pipeline ends with
"Query <id> is no longer active" error which causes task in workflow to fail.
I have also check option:
"Query <id> is no longer active" error which causes task in workflow to fail.
I have also check option:
.trigger(once=True) and query.processAllAvailable()
but the behavior is the same.
Does anybody knows why this error occurs? How to avoid it ?
but the behavior is the same.
Does anybody knows why this error occurs? How to avoid it ?
