I am consuming an IoT stream with thousands of different signals using Structured Streaming. During processing of the stream, I need to know the previous timestamp and value for each signal in the micro batch. The signal stream is eventually written to a delta table. Every signal is expected to be sent at least once every hour.
Is it possible to make use of the internal State Store as a cache to store this custom state of the previous timestamp and value for each signal?
If not, what would be the canonical approach to maintain such a state?
These are the approaches that I can think of.
Approach 1:
Perform a join on the stream with the target table itself to get the previous signal timestamp and value.
Approach 2:
Maintain a separate ‘state table’ containing the previous timestamp and value for each signal. The ‘state table’ would then be joined with the stream to get the previous signal timestamp and value.
On receiving new signal values, the ‘state table’ would be updated using merge into.
