cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

SamCallister
by New Contributor II
  • 13737 Views
  • 8 replies
  • 3 kudos

Dynamic Partition Overwrite for Delta Tables

Spark supports dynamic partition overwrite for parquet tables by setting the config: spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic") before writing to a partitioned table. With delta tables is appears you need to manually specif...

  • 13737 Views
  • 8 replies
  • 3 kudos
Latest Reply
alijen
New Contributor II
  • 3 kudos

@SamCallister wrote: Spark supports dynamic partition overwrite for parquet tables by setting the config:spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")before writing to a partitioned table. With delta tables is appears you need ...

  • 3 kudos
7 More Replies
Direo
by Contributor
  • 1031 Views
  • 3 replies
  • 6 kudos
  • 1031 Views
  • 3 replies
  • 6 kudos
Latest Reply
Kaniz
Community Manager
  • 6 kudos

Hi @Direo Direo​ , Does the above suggestion help you? Were you able to write tables to Delta Lake using upsert mode?

  • 6 kudos
2 More Replies
Vu_QuangNguyen
by New Contributor
  • 2193 Views
  • 0 replies
  • 0 kudos

Structured streaming from an overwrite delta path

Hi experts, I need to ingest data from an existing delta path to my own delta lake. The dataflow is as shown in the diagram: Data team reads full snapshot of a database table and overwrite to a delta path. This is done many times per day, but...

0693f000007OoRcAAK
  • 2193 Views
  • 0 replies
  • 0 kudos
Labels