Running large window spark structured streaming aggregations with small slide duration

[serg-v]

I want to run aggregations on large windows (90 days) with small slide duration (5 minutes).

Straightforward solution leads to giant state around hundreds of gigabytes, which doesn't look acceptable.

Is there any best practices doing this?

Now I consider following scenarios:

1. Use flatMapGroupsWithState and implement EWMA (exponentially weighted moving average) instead of average to reduce state. Is there good library for EWMA?
2. Somehow join data from two streams - e.g. 90 day window with 1 day slide and 1 day window with 5 minute slide

Any other ideas?

Thread in azure q&a

[Kaniz]

Hi @Sergey Volkov​, Thanks for your question. Here are some fantastic articles on EWMA and Event-time Aggregation in Apache Spark™’s Structured Streaming. Please have a look. Let us know if that helps.

• https://towardsdatascience.com/time-series-from-scratch-exponentially-weighted-moving-averages-ewma-...
• https://databricks.com/blog/2017/05/08/event-time-aggregation-watermarking-apache-sparks-structured-...
• https://medium.com/expedia-group-tech/apache-spark-structured-streaming-operations-5-of-6-40d907866f...
• https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html

[jose_gonzalez]

HI @Sergey Volkov​,

Just a friendly follow-up. Are you still looking for help or did any of the docs that Kaniz has shared help you?

[serg-v]

Hi.

> Are you still looking for help

No, thank you, we have implemented EWMA using flatMapGroupsWithState.

> did any of the docs that Kaniz has shared help you?

Not really. They are just slightly connected to my problem.