- 16248 Views
- 8 replies
- 3 kudos
Dynamic Partition Overwrite for Delta Tables
Spark supports dynamic partition overwrite for parquet tables by setting the config: spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic") before writing to a partitioned table. With delta tables is appears you need to manually specif...
- 16248 Views
- 8 replies
- 3 kudos
- 3 kudos
@SamCallister wrote: Spark supports dynamic partition overwrite for parquet tables by setting the config:spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")before writing to a partitioned table. With delta tables is appears you need ...
- 3 kudos
- 1581 Views
- 3 replies
- 6 kudos
- 1581 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi @Direo Direo​ , Does the above suggestion help you? Were you able to write tables to Delta Lake using upsert mode?
- 6 kudos
- 2568 Views
- 0 replies
- 0 kudos
Structured streaming from an overwrite delta path
Hi experts, I need to ingest data from an existing delta path to my own delta lake. The dataflow is as shown in the diagram: Data team reads full snapshot of a database table and overwrite to a delta path. This is done many times per day, but...
- 2568 Views
- 0 replies
- 0 kudos