https://community.databricks.com/t5/data-engineering/how-to-do-a-full-load-using-dlt-pipeline/m-p/110043#M43469 <P><SPAN>if I use "spark.readStream" it does incremental loads and If I do "spark.read" it creates a materialised view.</SPAN><BR /><BR /><SPAN>What I want is:&nbsp; do a full load each time(no need of scd types) and it should be a streaming table and not a materialised view.<BR /><BR />Any help would be appreciable.</SPAN></P> Wed, 12 Feb 2025 20:38:54 GMT ImranA 2025-02-12T20:38:54Z https://community.databricks.com/t5/data-engineering/how-to-do-a-full-load-using-dlt-pipeline/m-p/110043#M43469 <P><SPAN>if I use "spark.readStream" it does incremental loads and If I do "spark.read" it creates a materialised view.</SPAN><BR /><BR /><SPAN>What I want is:&nbsp; do a full load each time(no need of scd types) and it should be a streaming table and not a materialised view.<BR /><BR />Any help would be appreciable.</SPAN></P> Wed, 12 Feb 2025 20:38:54 GMT https://community.databricks.com/t5/data-engineering/how-to-do-a-full-load-using-dlt-pipeline/m-p/110043#M43469 ImranA 2025-02-12T20:38:54Z https://community.databricks.com/t5/data-engineering/how-to-do-a-full-load-using-dlt-pipeline/m-p/110070#M43479 <P>In Databricks Delta Live Tables (DLT), you can't directly truncate a streaming table, as streaming tables are append-only by design.&nbsp;</P><P>However in your scenario, you could possibly use a job workflow, where the first task runs a sql statement (using serverless sql) to truncate the table and the following job runs your DLT pipeline.</P> Thu, 13 Feb 2025 03:06:09 GMT https://community.databricks.com/t5/data-engineering/how-to-do-a-full-load-using-dlt-pipeline/m-p/110070#M43479 AmanSehgal 2025-02-13T03:06:09Z