topic Re: Dynamic Partition Overwrite for Delta Tables in Data Engineering

Dynamic Partition Overwrite for Delta Tables

SamCallister — Fri, 22 Nov 2019 21:06:32 GMT

Spark supports dynamic partition overwrite for parquet tables by setting the config:

spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")

before writing to a partitioned table. With delta tables is appears you need to manually specify which partitions you are overwriting with

replaceWhere

https://docs.databricks.com/delta/delta-batch.html#overwrite-using-dataframes

df.write
  .format("delta")
  .mode("overwrite")
  .option("replaceWhere", "date >= '2017-01-01' AND date <= '2017-01-31'")
  .save("/delta/events")

Is there anyway to overwrite partitions in a delta table without having to manually specify which partitions should be overwritten?

Re: Dynamic Partition Overwrite for Delta Tables

Anonymous — Thu, 12 Dec 2019 15:09:13 GMT

Is there an update on this? I've also noted that the configuration spark.sql.sources.partitionOverwriteMode does not affect delta tables. For what I can understand, to overwrite partitions dynamically we are stuck with spark.databricks.optimizer.dynamicPartitionPruning but only when used as join key.

It would be usefull to use .option("partitionOverwriteMode", "dynamic") for INSERT OVERWRITE statement

Re: Dynamic Partition Overwrite for Delta Tables

ChristianHofste — Mon, 04 May 2020 09:06:40 GMT

I am still waiting for an update on that...

Re: Dynamic Partition Overwrite for Delta Tables

Rajeswari — Tue, 09 Jun 2020 03:57:36 GMT

I am facing the same issue. Is there an update/suggested solution on how we could overwrite the Delta file partitions dynamically?

Re: Dynamic Partition Overwrite for Delta Tables

OndrejHavlicek — Fri, 21 Aug 2020 09:15:30 GMT

There seems to be some reluctance to implement this:

https://github.com/delta-io/delta/issues/348

But there is an open PR:

https://github.com/delta-io/delta/pull/371

Re: Dynamic Partition Overwrite for Delta Tables

MichaelDavidson — Wed, 16 Dec 2020 08:57:34 GMT

I think good will be for you to create a folder/partition for every FILE_DATE, because you may not need rewrite all files, only a specific file_date will have to be overwritten.

thetermpapers.org

Re: Dynamic Partition Overwrite for Delta Tables

OndrejHavlicek — Wed, 16 Dec 2020 10:16:13 GMT

I don't see how this answers the question.. Please do not spam in this forum with question-irrelevant links..

Re: Dynamic Partition Overwrite for Delta Tables

MikeLivshutz — Mon, 14 Nov 2022 01:38:58 GMT

Dynamic Partition Overwrite was implemented in Databricks 11.1

https://docs.databricks.com/delta/selective-overwrite.html#

Re: Dynamic Partition Overwrite for Delta Tables

alijen — Wed, 16 Aug 2023 09:15:27 GMT

@SamCallister wrote:

Spark supports dynamic partition overwrite for parquet tables by setting the config:
spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")
before writing to a partitioned table. With delta tables is appears you need to manually specify which partitions you are overwriting with
replaceWhere
thttps://docs.databricks.com/delta/delta-batch.html#overwrite-using-dataframes color blind test
df.write
  .format("delta")
  .mode("overwrite")
  .option("replaceWhere", "date >= '2017-01-01' AND date <= '2017-01-31'")
  .save("/delta/events")
Is there anyway to overwrite partitions in a delta table without having to manually specify which partitions should be overwritten?

I'm also messing around, it's hard to find tutorials. Help me