topic Re: Structured Streaming writeStream - Query is no longer active causes task to fail in Data Engineering

Structured Streaming writeStream - Query is no longer active causes task to fail

p_romm — Wed, 19 Feb 2025 11:26:30 GMT

Hi,

I execute readStream/writeStream in workflow task.
Write stream uses .trigger(availableNow=True) option. After writeStream I'm waiting query to finish with

query.awaitTermination(). However from time to time, pipeline ends with
"Query <id> is no longer active" error which causes task in workflow to fail.
I have also check option:

.trigger(once=True) and query.processAllAvailable()
but the behavior is the same.

Does anybody knows why this error occurs? How to avoid it ?

Re: Structured Streaming writeStream - Query is no longer active causes task to fail

Alberto_Umana — Wed, 19 Feb 2025 12:44:21 GMT

Hello @p_romm - Are you using serverless compute?

Re: Structured Streaming writeStream - Query is no longer active causes task to fail

p_romm — Mon, 24 Feb 2025 09:00:17 GMT

@Alberto_Umana yes, it is used on serveless, I didn't test it in general compute yet.
This is long running task in foreachBatch function, and in the messages I get also:
Query <id> has not received any updates in 60 seconds. Spark Session ID is <id>

Re: Structured Streaming writeStream - Query is no longer active causes task to fail

p_romm — Mon, 24 Feb 2025 10:34:40 GMT

However it looks like that 'Query <id> has not received any updates in 60 seconds' is not correlated with "Query <id> is no longer active"

Re: Structured Streaming writeStream - Query is no longer active causes task to fail

cmathieu — Mon, 28 Apr 2025 16:15:18 GMT

@Alberto_Umana this bug was apparently fixed a few months ago, but we're still facing the same issue on our end.