- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-13-2025 01:59 PM - edited 01-13-2025 01:59 PM
For this style of ETL, there are 2 methods.
The first method, strictly for partitioned tables, is Dynamic Partition Overwrites, which require a Spark configuration to be set and detect which partitions that are to be overwritten by scanning the input data.
The second method, replaceWhere, does not require the target table to be partitioned, but does require the user to explicitly tell the engine about what parts of the data to overwrite. Notably, you can scan the source dataset yourself and create the replaceWhere expressions "dynamically" yourself.
If you use overwrite mode and do not specify either of these options, the target table will be overwritten by the source dataframe, which for this workflow is likely not what you want.