AttributeError: 'DataFrame' object has no attribute 'dropDuplicatesWithinWatermark'

johanjohan — Mon, 19 Feb 2024 14:38:33 GMT

Hello,

I have some trouble deduplicating rows on the "id" column, with the method "dropDuplicatesWithinWatermark" in a pipeline. When I run this pipeline, I get the error message:

"AttributeError: 'DataFrame' object has no attribute 'dropDuplicatesWithinWatermark'"

Here is part of the code:

@dlt.table(

name="streaming_table",

comment="This table is used to test the drop duplicates with watermark"

)

def streaming_table_fct():

stream_df = spark.readStream.table("schema.table") \

.filter(f.col("kind") == "abc") \

.withWatermark("meta_created", "24 hours")

stream_df.dropDuplicatesWithinWatermark(["id"])

return stream_df

topic AttributeError: 'DataFrame' object has no attribute 'dropDuplicatesWithinWatermark' in Data Engineering

AttributeError: 'DataFrame' object has no attribute 'dropDuplicatesWithinWatermark'