Chaining stateful Operator

Maatari — Tue, 13 Aug 2024 12:53:47 GMT

I would like to do a groupby followed by a join in structured streaming. I would read from from two delta table in snapshot mode i.e. latest snapshot.

My question is specifically about chaining the stateful operator.

groupby is update mode

chaning groupby and join, must be append mode overall.

But does it means that the groupby would behave as if it was append as well, or the groupby can be in update mode and the join in append mode ?

Re: Chaining stateful Operator

mark_ott — Mon, 17 Nov 2025 11:46:13 GMT

When chaining stateful operators like groupBy (aggregation) and join in Spark Structured Streaming, there are specific rules about the output mode required for the overall query and the behavior of each operator.

Output Mode Requirements

The groupBy operator (stateful aggregation) supports update and complete output modes when used alone because it may update existing aggregated values as new data arrives.
The join between two streaming DataFrames must use append output mode overall, meaning only newly joined rows are emitted downstream.

Behavior When Chaining Operators

When you chain a groupBy (update mode) followed by a streaming join, the overall query is required to run in append mode because joins in Structured Streaming only support append output.
This does not mean that the groupBy operator itself shifts to append mode internally. The aggregation still behaves like an update aggregation: it maintains state and recalculates aggregates as new data arrives. However, Spark will output only the newly joined records, not updated aggregations, downstream—effectively discarding any updated rows not resulting in a new join.

topic Chaining stateful Operator in Data Engineering

Chaining stateful Operator

Re: Chaining stateful Operator

Output Mode Requirements

Behavior When Chaining Operators