I'd like to ingest data into my ADLS from sql server in an incremental manner using Delta Live Tables.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-01-2022 06:07 AM
I'd like to ingest data into my ADLS from sql server in an incremental manner using Delta Live Tables. I do not want to use any staging tables. I was using CDC, While I call dlt.apply_changes, its asking me to specify source and target. SInce source here is a DLT table, so I need to create a dlt table first (intermediate) by reading from sql server source and then use it as source and apply CDC functionality on that table and load data into target table. But isn't it like full load from source everytime to an intermediate table in ADLS and then load to target table using CDC ? How can we achieve it without having an intermediate table ?
Is there any other way we can achieve incremental in DLT from sql server source ?
- Labels:
-
Delta Live Tables
-
DLT