Structured Streaming writeStream - Query is no longer active causes task to fail

[p_romm]

Hi, 

I execute readStream/writeStream in workflow task. 
Write stream uses .trigger(availableNow=True) option. After writeStream I'm waiting query to finish with 

query.awaitTermination(). However from time to time, pipeline ends with 
"Query <id> is no longer active" error which causes task in workflow to fail. 
I have also check option:

.trigger(once=True)  and query.processAllAvailable()
but the behavior is the same. 

Does anybody knows why this error occurs? How to avoid it ? 

[Alberto_Umana]

Hello @p_romm  - Are you using serverless compute?

[p_romm]

@Alberto_Umana yes, it is used on serveless, I didn't test it in general compute yet. 
This is long running task in foreachBatch function,  and in the messages I get also:
Query <id> has not received any updates in 60 seconds. Spark Session ID is <id>

[p_romm]

However it looks like that 'Query <id> has not received any updates in 60 seconds' is not correlated with "Query <id> is no longer active"