- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-18-2025 05:15 PM
Hello @alexbush-mas
Good day!!
Unioning the streams is the standard method for feeding multiple input streams into a single transformWithStateInPandas operation for custom stream-stream joins, so your intuition is correct. After grouping the input by the join key (user_id), the example in the Databricks documentation https://learn.microsoft.com/en-gb/azure/databricks/stateful-applications/examples (under Example stateful applications) treats it as a single DataFrame; however, it specifically refers to this pattern for "joins across multiple streams." Prior to the groupBy and transform, the union takes place upstream.
I am trying to replicate the same but not sure the time line. I will give it a try by Monday.