Databricks Community

ImranA · 02-12-2025

if I use "spark.readStream" it does incremental loads and If I do "spark.read" it creates a materialised view.What I want is: do a full load each time(no need of scd types) and it should be a streaming table and not a materialised view.Any help woul...

ImranA · 12-12-2024

Hi,I am having an issue when I try to load the DLT bronze table for the first time, it goes through multiple parquet file using a loop(loop is just helping to provide the snapshot and version sequentially to "dlt.apply_changes_from_snapshot()"), And ...

ImranA · 12-09-2024

For example there is a table called "cars", if I remove the table from DLT pipeline and drop the table from catalogue. Now if I change the schema of the table, and create the table again using the same table name "cars" through the same pipeline, Why...

ImranA

can I read from a dlt.view using spark.read.table?

ImranA · 02-12-2025

For anyone else struggling with the same, I was overcomplicating this. basically next_snapshot runs as a loop itself. so you don't need to provide a separate loop for it to work.Please check the below code:def next_snapshot(latest_version): """ ...

ImranA · 02-12-2025

Hi @yvishal519 did you get to know how to do a full load because "spark.readStream" do incremental loads. If I do "spark.read" it creates a materialised view.What I want is: do a full load each time(no need of scd types) and it should be a streaming...

ImranA · 12-12-2024

@gchandracan you help me with the above?In summary, When I loop through the snapshots, the API doesn't create a history in the table that how the data is changing but only keeps the latest records from the most recent parquet file when I do full refr...

ImranA · 12-12-2024

By First Load I mean Full Refresh in DLT pipeline.

Databricks Community

User Stats

User Activity

How to do a Full Load using DLT pipeline

dlt.apply_changes_from_snapshot() is not picking up historical data

Schema issue when dropping a delta live table

Re: What is the difference between View and Table in Delta Live Table pipeline

Re: dlt.apply_changes_from_snapshot() is not picking up historical data

Re: Implementing Full Load Strategy with Delta Live Tables and Unity Catalog

Re: dlt.apply_changes_from_snapshot() is not picking up historical data

Re: dlt.apply_changes_from_snapshot() is not picking up historical data