Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-03-2025 12:20 PM
This should according to this blog post basically work, right? However, I'm getting the same error
Multiple Stateful Streaming Operators | Databricks Blog
Or am I missing something?
rate_df = spark.readStream.format("rate").option("rowsPerSecond", "1").load()
rate_df = rate_df.withWatermark("timestamp", "2 seconds")
# display(rate_df)
counts1 = rate_df.groupBy(F.window("timestamp", "10 seconds")).count()
counts2 = counts1.groupBy(F.window(F.window_time(counts1.window), "20 seconds")).count()
display(counts2)