Dynamic Partition Overwrite for Delta Tables
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-22-2019 01:06 PM
Spark supports dynamic partition overwrite for parquet tables by setting the config:
spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")
before writing to a partitioned table. With delta tables is appears you need to manually specify which partitions you are overwriting with
replaceWhere https://docs.databricks.com/delta/delta-batch.html#overwrite-using-dataframes
df.write
.format("delta")
.mode("overwrite")
.option("replaceWhere", "date >= '2017-01-01' AND date <= '2017-01-31'")
.save("/delta/events")
Is there anyway to overwrite partitions in a delta table without having to manually specify which partitions should be overwritten?
Labels:
- Labels:
-
Overwrite
-
Partitions