https://community.databricks.com/t5/data-engineering/i-d-like-to-ingest-data-into-my-adls-from-sql-server-in-an/m-p/19350#M12953 <P>I'd like to ingest data into my ADLS from sql server in an incremental manner using Delta Live Tables. I do not want to use any staging tables. I was using CDC, While I call dlt.apply_changes, its asking me to specify source and target. SInce source here is a DLT table, so I need to create a dlt table first (intermediate) by reading from sql server source and then use it as source and apply CDC functionality on that table and load data into target table. But isn't it like full load from source everytime to an intermediate table in ADLS and then load to target table using CDC ? How can we achieve it without having an intermediate table ?</P><P>Is there any other way we can achieve incremental in DLT from sql server source ?</P> Thu, 01 Dec 2022 14:07:12 GMT ravinchi 2022-12-01T14:07:12Z https://community.databricks.com/t5/data-engineering/i-d-like-to-ingest-data-into-my-adls-from-sql-server-in-an/m-p/19350#M12953 <P>I'd like to ingest data into my ADLS from sql server in an incremental manner using Delta Live Tables. I do not want to use any staging tables. I was using CDC, While I call dlt.apply_changes, its asking me to specify source and target. SInce source here is a DLT table, so I need to create a dlt table first (intermediate) by reading from sql server source and then use it as source and apply CDC functionality on that table and load data into target table. But isn't it like full load from source everytime to an intermediate table in ADLS and then load to target table using CDC ? How can we achieve it without having an intermediate table ?</P><P>Is there any other way we can achieve incremental in DLT from sql server source ?</P> Thu, 01 Dec 2022 14:07:12 GMT https://community.databricks.com/t5/data-engineering/i-d-like-to-ingest-data-into-my-adls-from-sql-server-in-an/m-p/19350#M12953 ravinchi 2022-12-01T14:07:12Z https://community.databricks.com/t5/data-engineering/i-d-like-to-ingest-data-into-my-adls-from-sql-server-in-an/m-p/19351#M12954 <P>Hey @Ravindra Ch​&nbsp;, even we have SAP sources and are loading currently to intermediate delta tables to move to target. I don't think there is any other way to directly move the data to target.</P> Fri, 02 Dec 2022 04:46:21 GMT https://community.databricks.com/t5/data-engineering/i-d-like-to-ingest-data-into-my-adls-from-sql-server-in-an/m-p/19351#M12954 Geeta1 2022-12-02T04:46:21Z https://community.databricks.com/t5/data-engineering/i-d-like-to-ingest-data-into-my-adls-from-sql-server-in-an/m-p/19353#M12956 <P>Thanks @Geeta Sai Boddu​&nbsp;for your suggestion. Can you share some thoughts on the approach you follwed in using CDC with an intermediate table to ingest tables dynamically if you don't mind? </P> Fri, 02 Dec 2022 15:58:28 GMT https://community.databricks.com/t5/data-engineering/i-d-like-to-ingest-data-into-my-adls-from-sql-server-in-an/m-p/19353#M12956 ravinchi 2022-12-02T15:58:28Z https://community.databricks.com/t5/data-engineering/i-d-like-to-ingest-data-into-my-adls-from-sql-server-in-an/m-p/19354#M12957 <P>If you have a CDC feed, looks like we can use this: <A href="https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-cdc.html" target="test_blank">https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-cdc.html</A></P> Thu, 26 Jan 2023 15:59:49 GMT https://community.databricks.com/t5/data-engineering/i-d-like-to-ingest-data-into-my-adls-from-sql-server-in-an/m-p/19354#M12957 Sandeep 2023-01-26T15:59:49Z https://community.databricks.com/t5/data-engineering/i-d-like-to-ingest-data-into-my-adls-from-sql-server-in-an/m-p/19355#M12958 <P>In our case we are trying since some weeks ago implementing a data flow ingestion pipeline via DLT which is currently done in other technologies.</P><P></P><P>We are not succeeding in use CDC. It looks to be very limited and the source data for the APPLY CHANGES INTO function doesn't allow UNIONs, some kind of JOINs, doing any intermediate action with side effects... I am thinking about posting a question here but first will investigate and see if I find any help.</P><P></P><P>Our issue resides in trying to delete gold before inserting but there's apparently no way.</P> Mon, 30 Jan 2023 17:02:26 GMT https://community.databricks.com/t5/data-engineering/i-d-like-to-ingest-data-into-my-adls-from-sql-server-in-an/m-p/19355#M12958 Llop 2023-01-30T17:02:26Z