Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-16-2023 02:15 AM
@SamCallister wrote:
Spark supports dynamic partition overwrite for parquet tables by setting the config:
spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")before writing to a partitioned table. With delta tables is appears you need to manually specify which partitions you are overwriting with
replaceWherethttps://docs.databricks.com/delta/delta-batch.html#overwrite-using-dataframescolor blind test
df.write .format("delta") .mode("overwrite") .option("replaceWhere", "date >= '2017-01-01' AND date <= '2017-01-31'") .save("/delta/events")Is there anyway to overwrite partitions in a delta table without having to manually specify which partitions should be overwritten?
I'm also messing around, it's hard to find tutorials. Help me