https://community.databricks.com/t5/data-engineering/how-to-implement-incremental-loading-in-azure-databricks-for-etl/m-p/119929#M45995 <P>Hi everyone,</P><P>I'm currently working on an ETL process using Azure Databricks (Standard Tier) where I load data from Azure SQL Database into Databricks. I run a notebook daily to extract, transform, and load the data for Power BI reports.</P><P>Right now, the notebook loads all data from the beginning every time it runs, which is inefficient and causes unnecessary processing time. I want to switch to incremental loading, so the job only fetches new or changed records since the last successful run.</P><P>My setup:</P><UL><LI>Source: Azure SQL Database</LI><LI>Target: Databricks Delta Table</LI><LI>Scheduler: Daily Databricks job</LI><LI>Purpose: Power BI dashboards using processed data</LI></UL><P>What I'm looking for:</P><UL><LI>A standard or recommended approach to implement incremental loading in Databricks</LI><LI>Best practices for tracking the last load timestamp (e.g., using a watermark)</LI><LI>Example code or a step-by-step tutorial</LI><LI>Any built-in Databricks utilities or patterns to support this on the Standard Tier</LI></UL><P>If you've set this up before or know of any good resources, I’d really appreciate your help!</P><P>Thanks in advance!</P> Thu, 22 May 2025 06:09:29 GMT chexa_Wee 2025-05-22T06:09:29Z https://community.databricks.com/t5/data-engineering/how-to-implement-incremental-loading-in-azure-databricks-for-etl/m-p/119929#M45995 <P>Hi everyone,</P><P>I'm currently working on an ETL process using Azure Databricks (Standard Tier) where I load data from Azure SQL Database into Databricks. I run a notebook daily to extract, transform, and load the data for Power BI reports.</P><P>Right now, the notebook loads all data from the beginning every time it runs, which is inefficient and causes unnecessary processing time. I want to switch to incremental loading, so the job only fetches new or changed records since the last successful run.</P><P>My setup:</P><UL><LI>Source: Azure SQL Database</LI><LI>Target: Databricks Delta Table</LI><LI>Scheduler: Daily Databricks job</LI><LI>Purpose: Power BI dashboards using processed data</LI></UL><P>What I'm looking for:</P><UL><LI>A standard or recommended approach to implement incremental loading in Databricks</LI><LI>Best practices for tracking the last load timestamp (e.g., using a watermark)</LI><LI>Example code or a step-by-step tutorial</LI><LI>Any built-in Databricks utilities or patterns to support this on the Standard Tier</LI></UL><P>If you've set this up before or know of any good resources, I’d really appreciate your help!</P><P>Thanks in advance!</P> Thu, 22 May 2025 06:09:29 GMT https://community.databricks.com/t5/data-engineering/how-to-implement-incremental-loading-in-azure-databricks-for-etl/m-p/119929#M45995 chexa_Wee 2025-05-22T06:09:29Z https://community.databricks.com/t5/data-engineering/how-to-implement-incremental-loading-in-azure-databricks-for-etl/m-p/120020#M46027 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/155027">@chexa_Wee</a>, you can leverage DLT feature to do so.&nbsp;</P> <P>Please check:&nbsp;<A href="https://docs.databricks.com/aws/en/dlt/transform" target="_blank">https://docs.databricks.com/aws/en/dlt/transform</A></P> <P><A href="https://docs.databricks.com/aws/en/dlt/stateful-processing" target="_blank">https://docs.databricks.com/aws/en/dlt/stateful-processing</A></P> <P>Here is the s<SPAN>tep-by-step tutorial:</SPAN>&nbsp;<A href="https://docs.databricks.com/aws/en/dlt/tutorials" target="_blank">https://docs.databricks.com/aws/en/dlt/tutorials</A></P> <P>&nbsp;</P> Fri, 23 May 2025 05:12:37 GMT https://community.databricks.com/t5/data-engineering/how-to-implement-incremental-loading-in-azure-databricks-for-etl/m-p/120020#M46027 nikhilj0421 2025-05-23T05:12:37Z