Spark structured streaming- calculate signal, help required! 🙏

[Galih]

Hello everyone!

I’m very very new to Spark Structured Streaming, and not a data engineer 😅I would appreciate guidance on how to efficiently process streaming data and emit only changed aggregate results over multiple time windows.

Input Stream:

Source: Amazon Kinesis

Microbatch granularity : Every 60 seconds

Schema:

(profile_id, gti, event_timestamp, event_type)

Where:

event_type ∈ { select, highlight, view }

Time Windows:

We need to maintain counts for rolling aggregates of the following windows:

1 hour

12 hours

24 hours

Output Requirement:

For each (profile_id, gti) combination, I want to emit only the current counts that changed during the current micro-batch.

The output record should look like this:

{

"profile_id": "profileid",

"gti": "amz1.gfgfl",

"select_count_1d": 5,

"select_count_12h": 2,

"select_count_1h": 1,

"highlight_count_1d": 20,

"highlight_count_12h": 10,

"highlight_count_1h": 3,

"view_count_1d": 40,

"view_count_12h": 30,

"view_count_1h": 3

}

Key Requirements:

Per key output: (profile_id, gti)

Emit only changed rows in the current micro-batch

This data is written to a feature store, so we want to avoid rewriting unchanged aggregates

Each emitted record should represent the latest counts for that key

What We Tried:

We implemented sliding window aggregations using groupBy(window()) for each time window. For example:

groupBy(

profile_id,

gti,

window(event_timestamp, windowDuration, "1 minute")

)

Spark didn’t allow joining those three streams for outer join limitation error between streams.

We tried to work around it by writing each stream to the memory and take a snapshot every 60 seconds but it does not only output the changed rows..

How would you go about this problem? Should we maintain three rolling time windows like we tried and find a way to join them or is there any other way you could think of?

Very lost here, any help would be very appreciated!!