Khaja_Zaffer
Esteemed Contributor

Hello @alexbush-mas 

Good day!! 

Unioning the streams is the standard method for feeding multiple input streams into a single transformWithStateInPandas operation for custom stream-stream joins, so your intuition is correct. After grouping the input by the join key (user_id), the example in the Databricks documentation https://learn.microsoft.com/en-gb/azure/databricks/stateful-applications/examples (under Example stateful applications) treats it as a single DataFrame; however, it specifically refers to this pattern for "joins across multiple streams." Prior to the groupBy and transform, the union takes place upstream.

I am trying to replicate the same but not sure the time line. I will give it a try by Monday.