cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Deleting Lakeflow pipelines impact on objects within.

mdungey
New Contributor II

I've seen hidden in some forums that Databricks are working on a fix so that when you delete a LDP pipeline it doesn't delete the underlying objects(streaming tables, mat views etc..).  Can anyone from an official source confirm this and maybe give some rough dates as to when this may become available.  I've heard Jan 2026 as a date but would like this confirmed with the PM for the project.

Until this is released it's pretty much unusable for us.  It carries far too much risk and as I understand there's no real way to mitigate this from happening.

3 REPLIES 3

Raman_Unifeye
Contributor III

nothing I came across on official docs. 

its a different behavious listed on DB document, I dont think you are looking for this one.

Behavior change when dataset definitions are removed from Lakeflow Spark Declarative Pipelines

An upcoming release of Lakeflow Spark Declarative Pipelines will change the behavior when a materialized view or streaming table is removed from a pipeline. With this change, the removed materialized view or streaming table will not be deleted automatically when the next pipeline update runs. Instead, you will be able to use the DROP MATERIALIZED VIEW command to delete a materialized view or the DROP TABLE command to delete a streaming table. After dropping an object, running a pipeline update will not recover the object automatically. A new object is created if a materialized view or streaming table with the same definition is re-added to the pipeline. You can, however, recover an object using the UNDROP command.


RG #Driving Business Outcomes with Data Intelligence

mdungey
New Contributor II

This was a step in the right direction but it's still the case if the whole pipeline is deleted the contained objects are also dropped.  Found this on reddit but not confident how official this is.

https://www.reddit.com/r/databricks/comments/1niczj7/comment/nm1ybfd/?context=3

 

Raman_Unifeye
Contributor III

yes, I would take that as a pinch of salt 😄


RG #Driving Business Outcomes with Data Intelligence