topic how to compute difference over time of a spark structure streaming? in Data Engineering

how to compute difference over time of a spark structure streaming?

chanansh — Wed, 18 Jan 2023 15:48:14 GMT

I have a table with a timestamp column (t) and a list of columns for which I would like to compute the difference over time (v), by some key(k): v_diff(t) = v(t)-v(t-1) for each k independently.

Normally I would write:

lag_window = Window.partitionBy(COLS_TO_DIFF).orderBy('timestamp')
 
for col in COLS_TO_DIFF:
  df = df.withColumn(
    col + "_diff",
    df[col] - F.lag(df[col]).over(lag_window))

But

AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets;

So, my question is how do I compute what I need?

Re: how to compute difference over time of a spark structure streaming?

chanansh — Sun, 22 Jan 2023 11:54:55 GMT

I asked it also in Stack-overflow since I don't get an answer here https://stackoverflow.com/questions/75161849/spark-structure-streaming-differentiate-over-time

Re: how to compute difference over time of a spark structure streaming?

chanansh — Wed, 08 Feb 2023 13:32:54 GMT

I found this but could not make it work https://www.databricks.com/blog/2022/10/18/python-arbitrary-stateful-processing-structured-streaming.html