pyspark.sql.utils.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-20-2023 01:57 AM
pyspark.sql.utils.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets
Getting this error while writing can any one please tell how we can resolve it
- Labels:
-
Windows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-21-2023 06:34 AM
Please give us the whole code by that we can understand what exactly you are doing
or you can connect with me in my LinkedIN and we can setup meeting.My LinkedIn URL is in my profile
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-21-2023 08:57 AM
I'm trying to run query on some table and then storing that result in some table .
query = stream
.writeStream
.format("delta")
.foreachBatch(batch_function) \
.option('checkpointLocation', self.checkpoint_loc)
.trigger(processingTime=self.trigger_interval)
And in that query I'm first using union function and then using window function. Their I'm facing this error pyspark.sql.utils.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-30-2025 09:07 PM
Hi @Gaurav_784295 ,
In Spark, In case of streaming, please use a time based column in window function. Because, In streaming we cant say "last 10 rows", "limit 10" etc. Because streaming never ends. So when you use window, please dont use columns like item_id or anything. Please make sure its event_time or any time based column.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-02-2026 07:06 AM