I'd like to ingest data into my ADLS from sql server in an incremental manner using Delta Live Tables.

[ravinchi]

I'd like to ingest data into my ADLS from sql server in an incremental manner using Delta Live Tables. I do not want to use any staging tables. I was using CDC, While I call dlt.apply_changes, its asking me to specify source and target. SInce source here is a DLT table, so I need to create a dlt table first (intermediate) by reading from sql server source and then use it as source and apply CDC functionality on that table and load data into target table. But isn't it like full load from source everytime to an intermediate table in ADLS and then load to target table using CDC ? How can we achieve it without having an intermediate table ?

Is there any other way we can achieve incremental in DLT from sql server source ?

[Geeta1]

Hey @Ravindra Ch​ , even we have SAP sources and are loading currently to intermediate delta tables to move to target. I don't think there is any other way to directly move the data to target.

[ravinchi]

Thanks @Geeta Sai Boddu​ for your suggestion. Can you share some thoughts on the approach you follwed in using CDC with an intermediate table to ingest tables dynamically if you don't mind?

[Kaniz]

Hi @Ravindra Ch​​, We haven’t heard from you since the last response from @Geeta Sai Boddu​, and I was checking back to see if their suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

[Sandeep]

If you have a CDC feed, looks like we can use this: https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-cdc.html

[Llop]

In our case we are trying since some weeks ago implementing a data flow ingestion pipeline via DLT which is currently done in other technologies.

We are not succeeding in use CDC. It looks to be very limited and the source data for the APPLY CHANGES INTO function doesn't allow UNIONs, some kind of JOINs, doing any intermediate action with side effects... I am thinking about posting a question here but first will investigate and see if I find any help.

Our issue resides in trying to delete gold before inserting but there's apparently no way.