fperry
New Contributor III

This should according to this blog post basically work, right? However, I'm getting the same error
Multiple Stateful Streaming Operators | Databricks Blog

Or am I missing something?

 

rate_df = spark.readStream.format("rate").option("rowsPerSecond", "1").load()

rate_df = rate_df.withWatermark("timestamp", "2 seconds")

# display(rate_df)

counts1 = rate_df.groupBy(F.window("timestamp", "10 seconds")).count()

counts2 = counts1.groupBy(F.window(F.window_time(counts1.window), "20 seconds")).count()

display(counts2)