What is the recommended approach for handling deletes in a Delta table?
I have a table in MySQL (no soft delete flag) that I read and write into Azure as a Delta table. My current flow is:
- If an ID exists in both MySQL and the Delta table → update the record in Delta.
- If an ID exists in MySQL but not in Delta → insert it into Delta.
The challenge is: how do I handle deletes? Specifically, if an ID exists in the Delta table but does not exist in the latest MySQL extract, I want to remove it from Delta. What’s the best way to implement this logic?