alijen
New Contributor II

@SamCallister wrote:

 

Spark supports dynamic partition overwrite for parquet tables by setting the config:

spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")

before writing to a partitioned table. With delta tables is appears you need to manually specify which partitions you are overwriting with

replaceWhere

thttps://docs.databricks.com/delta/delta-batch.html#overwrite-using-dataframescolor blind test

df.write
  .format("delta")
  .mode("overwrite")
  .option("replaceWhere", "date >= '2017-01-01' AND date <= '2017-01-31'")
  .save("/delta/events")

Is there anyway to overwrite partitions in a delta table without having to manually specify which partitions should be overwritten?


I'm also messing around, it's hard to find tutorials. Help me