Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-13-2025 04:00 AM
My suggestion wud be to try using MERGE INTO for delta tables which works with connector then using delete/insert statements. This will also keep your code in SQL as you wanted. your tables are not large so this shud be sufficient otherwise we need to consider spark/delta combination.
e.g.
merge_sql = f"""
MERGE INTO your_table AS target
USING (SELECT * FROM VALUES {values} AS t(id, col1, col2)) AS source
ON target.id = source.id
WHEN MATCHED THEN UPDATE SET col1 = source.col1, col2 = source.col2
WHEN NOT MATCHED THEN INSERT (id, col1, col2) VALUES (source.id, source.col1, source.col2)
"""
