Hubert-Dudek
Databricks MVP

I would implement stateful streaming by using transformWithStateInPandas to keep the state and implement the logic there.

I would avoid doing stream-stream JOINs.


My blog: https://databrickster.medium.com/