Walter_C
Databricks Employee
Databricks Employee

When using the AvailableNow trigger in Spark Structured Streaming, the behavior during a query failure is as follows:

  1. End Offset: The initial end offset set by the AvailableNow trigger does not change due to a query failure. The AvailableNow trigger processes all available data up to a specific point in time, and this end offset remains fixed even if the query fails.

  2. Query Resumption: If checkpointing is enabled, the query will resume from where it left off upon recovery. This means that the processing will continue from the last successfully processed offset, not from the beginning. The end offset remains the same as initially set by the AvailableNow trigger.

  3. Failure vs. End of Query: Spark Structured Streaming does differentiate between a query failure and the end of the query. A failure means the query did not complete successfully, and upon recovery, it will continue processing from the last checkpoint. The end of the query, in the context of AvailableNow, means that all data up to the specified end offset has been processed.

In summary, the end offset set by the AvailableNow trigger remains unchanged during a query failure, and the query will resume from the last checkpointed position upon recovery.