- 18032 Views
- 8 replies
- 3 kudos
Dynamic Partition Overwrite for Delta Tables
Spark supports dynamic partition overwrite for parquet tables by setting the config: spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic") before writing to a partitioned table. With delta tables is appears you need to manually specif...
- 18032 Views
- 8 replies
- 3 kudos
- 3 kudos
@SamCallister wrote: Spark supports dynamic partition overwrite for parquet tables by setting the config:spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")before writing to a partitioned table. With delta tables is appears you need ...
- 3 kudos
- 1841 Views
- 1 replies
- 5 kudos
- 1841 Views
- 1 replies
- 5 kudos
- 5 kudos
@Direo Direo​ , Yes, you use Merge syntax for that https://docs.delta.io/latest/delta-update.html.And is more efficient than overwriting if you want to update only part of the data, but you need to think about the logic of what to update so overwriti...
- 5 kudos
- 2784 Views
- 0 replies
- 0 kudos
Structured streaming from an overwrite delta path
Hi experts, I need to ingest data from an existing delta path to my own delta lake. The dataflow is as shown in the diagram: Data team reads full snapshot of a database table and overwrite to a delta path. This is done many times per day, but...
- 2784 Views
- 0 replies
- 0 kudos