Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-24-2022 06:23 AM
Yes, "Append" mode is doing a Upsert/Merge in pyspark, which is the same thing as the SQL Merge Into I believe; that's what gave me the above error.
And I had another version that did a Merge-Into by individual data source, but I spoke to Databricks and they recommended against doing that as I'd need to have sequential merges from each source to avoid the writes conflicting, and an error in one would result in a failure across them all.