Stream-stream window join after time window aggregation not working in 13.1

azera — Thu, 15 Jun 2023 07:59:44 GMT

Hey,

I'm trying to perform Time window aggregation in two different streams followed by stream-stream window join described here. I'm running Databricks Runtime 13.1, exactly as advised.

However, when I'm reproducing the following code:

clicksWindow = clicksWithWatermark.groupBy(
  clicksWithWatermark.clickAdId,
  window(clicksWithWatermark.clickTime, "1 hour")
).count()
 
impressionsWindow = impressionsWithWatermark.groupBy(
  impressionsWithWatermark.impressionAdId,
  window(impressionsWithWatermark.impressionTime, "1 hour")
).count()
 
clicksWindow.join(impressionsWindow, "window", "inner")

I'm not getting any result from the joined table in append mode. It is just empty no matter whether I'm using AdId in groupBy or not. The same behaviour is in Python and Scala.

If I join on window.end, not window, then I start receiving results but then I can use only inner join (as the joined condition, window.end, is not a watermarked column) but I need do to outer join for my use case (even with inner join, state seems to increase indefinitely).

Any help with reproducing this example is appreciated

Re: Stream-stream window join after time window aggregation not working in 13.1

Anonymous — Sun, 18 Jun 2023 12:57:32 GMT

Hi @Andrzej Zera

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

Re: Stream-stream window join after time window aggregation not working in 13.1

Happyfield7 — Fri, 27 Oct 2023 13:05:34 GMT

Hey,

I'm currently facing the same problem, so I would to know if you've made any progress in resolving this issue.

topic Stream-stream window join after time window aggregation not working in 13.1 in Data Engineering

Stream-stream window join after time window aggregation not working in 13.1

Re: Stream-stream window join after time window aggregation not working in 13.1

Re: Stream-stream window join after time window aggregation not working in 13.1