Maintaining Custom State in Structured Streaming

Starki — Wed, 29 Mar 2023 13:35:43 GMT

I am consuming an IoT stream with thousands of different signals using Structured Streaming. During processing of the stream, I need to know the previous timestamp and value for each signal in the micro batch. The signal stream is eventually written to a delta table. Every signal is expected to be sent at least once every hour.

Is it possible to make use of the internal State Store as a cache to store this custom state of the previous timestamp and value for each signal?

If not, what would be the canonical approach to maintain such a state?

These are the approaches that I can think of.

Approach 1:

Perform a join on the stream with the target table itself to get the previous signal timestamp and value.

Approach 2:

Maintain a separate ‘state table’ containing the previous timestamp and value for each signal. The ‘state table’ would then be joined with the stream to get the previous signal timestamp and value.

On receiving new signal values, the ‘state table’ would be updated using merge into.

Re: Maintaining Custom State in Structured Streaming

Soma — Sat, 08 Apr 2023 17:34:31 GMT

@Suteja Kanuri

Tried the above on streaming DF

But facing the below error

AttributeError: 'DataFrame' object has no attribute 'groupByKey'

Can you please let me know DBR runtime

topic Maintaining Custom State in Structured Streaming in Data Engineering

Maintaining Custom State in Structured Streaming

Re: Maintaining Custom State in Structured Streaming