Re-running DLT Pipeline Does Not Add Data After Delete

dbdude — Fri, 01 Sep 2023 18:54:08 GMT

I am using DLT and unity catalog and using managed tables. The first table in this pipeline is a live streaming table. I first did this in the SQL editor:

DELETE FROM my_table;

This appears to have deleted all the records, which I wanted since now when I do a SELECT, it shows 0 rows. Then I re-ran the pipeline. This first table is running this command:

CREATE OR REFRESH STREAMING LIVE TABLE my_table
COMMENT "Data from S3"
TBLPROPERTIES ("myCompanyPipeline.quality" = "bronze")
AS
SELECT * FROM cloud_files("s3://my-bucket/path/", "parquet");

However when this pipeline finishes, the table has no data in it. What am I not understanding?

Re: Re-running DLT Pipeline Does Not Add Data After Delete

BilalAslamDbrx — Tue, 05 Sep 2023 13:06:39 GMT

@Mo is correct!

topic Re-running DLT Pipeline Does Not Add Data After Delete in Data Engineering

Re-running DLT Pipeline Does Not Add Data After Delete

Re: Re-running DLT Pipeline Does Not Add Data After Delete