cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Re-running DLT Pipeline Does Not Add Data After Delete

dbdude
New Contributor II

I am using DLT and unity catalog and using managed tables. The first table in this pipeline is a live streaming table. I first did this in the SQL editor:

DELETE FROM my_table;

This appears to have deleted all the records, which I wanted since now when I do a SELECT, it shows 0 rows. Then I re-ran the pipeline. This first table is running this command:

CREATE OR REFRESH STREAMING LIVE TABLE my_table
COMMENT "Data from S3"
TBLPROPERTIES ("myCompanyPipeline.quality" = "bronze")
AS
SELECT * FROM cloud_files("s3://my-bucket/path/", "parquet");

However when this pipeline finishes, the table has no data in it. What am I not understanding?

1 REPLY 1

BilalAslamDbrx
Honored Contributor II
Honored Contributor II

@Mo is correct!

 

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.