12-01-2022 06:07 AM
I'd like to ingest data into my ADLS from sql server in an incremental manner using Delta Live Tables. I do not want to use any staging tables. I was using CDC, While I call dlt.apply_changes, its asking me to specify source and target. SInce source here is a DLT table, so I need to create a dlt table first (intermediate) by reading from sql server source and then use it as source and apply CDC functionality on that table and load data into target table. But isn't it like full load from source everytime to an intermediate table in ADLS and then load to target table using CDC ? How can we achieve it without having an intermediate table ?
Is there any other way we can achieve incremental in DLT from sql server source ?
12-01-2022 08:46 PM
Hey @Ravindra Ch , even we have SAP sources and are loading currently to intermediate delta tables to move to target. I don't think there is any other way to directly move the data to target.
12-02-2022 07:58 AM
Thanks @Geeta Sai Boddu for your suggestion. Can you share some thoughts on the approach you follwed in using CDC with an intermediate table to ingest tables dynamically if you don't mind?
12-02-2022 04:18 AM
Hi @Ravindra Ch, We haven’t heard from you since the last response from @Geeta Sai Boddu, and I was checking back to see if their suggestions helped you.
Or else, If you have any solution, please share it with the community, as it can be helpful to others.
Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.
01-26-2023 07:59 AM
If you have a CDC feed, looks like we can use this: https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-cdc.html
01-30-2023 09:02 AM
In our case we are trying since some weeks ago implementing a data flow ingestion pipeline via DLT which is currently done in other technologies.
We are not succeeding in use CDC. It looks to be very limited and the source data for the APPLY CHANGES INTO function doesn't allow UNIONs, some kind of JOINs, doing any intermediate action with side effects... I am thinking about posting a question here but first will investigate and see if I find any help.
Our issue resides in trying to delete gold before inserting but there's apparently no way.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group