https://community.databricks.com/t5/data-engineering/incremental-load-from-a-sql-table/m-p/111010#M43761 <P>hi,</P><P>You can use ADF to extract data incrementally into the bronze layer, and use DLT to load this data into silver with SCD1 or SCD2.<BR /><BR />You will need to store somewhere information about last extracted ID and read it in ADF pipeline to get only new records.</P> Mon, 24 Feb 2025 08:47:35 GMT MariuszK 2025-02-24T08:47:35Z https://community.databricks.com/t5/data-engineering/incremental-load-from-a-sql-table/m-p/110999#M43758 <P>Hi, Is there a way to read a sql table using jdbc as a streaming source? If not what is the best approach to read incremental data from a sql table and implement SCD Type 1 while loading into silver and gold layer. Can this be implemented using dlt pipeline?</P> Mon, 24 Feb 2025 02:59:01 GMT https://community.databricks.com/t5/data-engineering/incremental-load-from-a-sql-table/m-p/110999#M43758 AKon 2025-02-24T02:59:01Z https://community.databricks.com/t5/data-engineering/incremental-load-from-a-sql-table/m-p/111010#M43761 <P>hi,</P><P>You can use ADF to extract data incrementally into the bronze layer, and use DLT to load this data into silver with SCD1 or SCD2.<BR /><BR />You will need to store somewhere information about last extracted ID and read it in ADF pipeline to get only new records.</P> Mon, 24 Feb 2025 08:47:35 GMT https://community.databricks.com/t5/data-engineering/incremental-load-from-a-sql-table/m-p/111010#M43761 MariuszK 2025-02-24T08:47:35Z