topic Re: Running large window spark structured streaming aggregations with small slide duration in Data Engineering

Running large window spark structured streaming aggregations with small slide duration

serg-v — Fri, 17 Jun 2022 09:25:53 GMT

I want to run aggregations on large windows (90 days) with small slide duration (5 minutes).

Straightforward solution leads to giant state around hundreds of gigabytes, which doesn't look acceptable.

Is there any best practices doing this?

Now I consider following scenarios:

Use flatMapGroupsWithState and implement EWMA (exponentially weighted moving average) instead of average to reduce state. Is there good library for EWMA?
Somehow join data from two streams - e.g. 90 day window with 1 day slide and 1 day window with 5 minute slide

Any other ideas?

jose_gonzalez — Fri, 29 Jul 2022 20:13:41 GMT

HI @Sergey Volkov,

Just a friendly follow-up. Are you still looking for help or did any of the docs that Kaniz has shared help you?

serg-v — Sun, 21 Aug 2022 09:51:07 GMT

Hi.

> Are you still looking for help

No, thank you, we have implemented EWMA using flatMapGroupsWithState.

> did any of the docs that Kaniz has shared help you?

Not really. They are just slightly connected to my problem.