pyspark.sql.utils.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets

[Gaurav_784295]

pyspark.sql.utils.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets

Getting this error while writing can any one please tell how we can resolve it

[Aviral-Bhardwaj]

Please give us the whole code by that we can understand what exactly you are doing

or you can connect with me in my LinkedIN and we can setup meeting.My LinkedIn URL is in my profile

[Gaurav_784295]

I'm trying to run query on some table and then storing that result in some table .

query = stream

.writeStream

.format("delta")

.foreachBatch(batch_function) \

.option('checkpointLocation', self.checkpoint_loc)

.trigger(processingTime=self.trigger_interval)

And in that query I'm first using union function and then using window function. Their I'm facing this error pyspark.sql.utils.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets

[preetmdata]

Hi @Gaurav_784295  ,

In Spark, In case of streaming, please use a time based column in window function. Because, In streaming we cant say "last 10 rows", "limit 10" etc. Because streaming never ends. So when you use window, please dont use columns like item_id or anything. Please make sure its event_time or any time based column.

[siva-anantha]

I share the same perspective as @preetmdata on this