Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-03-2025 06:27 AM
hi @jhonm_839
thanks for your response to the post. I appreciate your time.
I have watermark on dataframe in palce and used last(), min() etc.. aggregate functions its still filing.
final_df = df.withWatermark("kafka_ts", "1 hour").withColumn(
"s_featureno",
when(
col("event_type") == "WAYB",
col("s_featureno")
).otherwise(
last("s_featureno", ignorenulls=True).over(Window.partitionBy("sno", "s_nr", "SAKey").orderBy("kafka_ts"))
)
)
still getting same error, can you please help me here.
