topic pyspark dropDuplicates performance issue in Data Engineering

pyspark dropDuplicates performance issue

sanjay — Mon, 12 Feb 2024 05:26:16 GMT

Hi,

I am trying to delete duplicate records found by key but its very slow. Its continuous running pipeline so data is not that huge but still it takes time to execute this command.

df = df.dropDuplicates(["fileName"])

Is there any better approach to delete duplicate data from pyspark dataframe.

Regards,

Sanjay

Re: pyspark dropDuplicates performance issue

sanjay — Mon, 12 Feb 2024 08:47:21 GMT

Thank you @Retired_mod. As I am trying to remove duplicate only on single column, so am specifying column name in dropDuplicates. Still its very slow. Can you provide more context on last point i.e.

Streamlining Your Data with Grouping and Aggregation: To easily condense your dataset by a single column's values, utilize the power of aggregation functions.

Is there any possibility to tune dropDuplicate

Re: pyspark dropDuplicates performance issue

NandiniN — Sat, 01 Feb 2025 07:19:04 GMT

Before dropDuplicates eensure that your DataFrame operations are optimized by caching intermediate results if they are reused multiple times. This can help reduce the overall execution time.

We could use some aggregates and grouping like

df_deduped = df.groupBy("fileName").agg(first("fileName").alias("fileName"))