Before applying the window function, try repartitioning your DataFrame based on the key (or the salted key). This can help distribute the data more evenly across the executors.

from pyspark.sql import Window
from pyspark.sql.functions import lag

# Repartition DataFrame
df = df.repartition("key")

# Define window specification
window_spec = Window.partitionBy("key").orderBy("eff_date")

# Add previous end date
df = df.withColumn("prev_end_date", lag("end_date", 1).over(window_spec))

# Show the result
df.show()