- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-27-2025 12:59 PM
I would need to dive deeper in your scenario but it sounds to me a strategy could be:
1) Create a view in your SQL Server database with "current data" UNION "historical data". You can set an additional boolean field with True in first query and False in second query.
2) Create an external federated catalog to your SQL Server to be able to run this query OR simply use SQL Server Spark access connector from your pipeline. Try to limit retrieved rows based on pre-detection of unchanged rows.
3) Design and create a target delta table matching the previous query along with any other field that makes sense in your scenario
3) In the pipeline use this query to create a dataframe and use MERGE command to insert, update or delete target rows in your destination delta table. Consider whether you need additional fields in your source to set each row as Inderted, updated or deleted. Otherwise, set a default policy to always insert if not present, update if present (easy part) and concerning "delete", it will depend on how you implement this CDC source!
This is only a high level approach. I hope it helps.