cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Change in DLT Pipeline Deletion Behavior associated Tables Not Being Deleted

surajitDE
Contributor

Hi Team,

We have noticed a change in the behavior of our Lakeflow pipelines.

Previously, when we deleted a DLT pipeline, the associated resources, such as materialized views and streaming tables, were also deleted along with the pipeline.

However, recently, when we delete a pipeline, the associated materialized views and streaming tables remain in Unity Catalog and are not automatically deleted.

Could you please confirm:

  1. Has there been any recent update or change in the pipeline deletion behavior?
  2. Is this now the expected behavior?
  3. Is there any configuration or recommended approach to automatically clean up the associated materialized views and streaming tables when a pipeline is deleted?

This is creating additional manual effort, as we now need to identify and delete the associated resources separately.

Surajit Metya
1 ACCEPTED SOLUTION

Accepted Solutions

juanlozadab
Contributor

Hi @surajitDE ,

Yes, this changed, and what you are seeing is the new intended behaviour.

The default flipped on September 25, 2026. From the release notes: when you
delete a Unity Catalog pipeline that uses the default publishing mode, the UI
and the Pipelines DELETE API retain the associated materialized views,
streaming tables and views by default. So nothing is broken on your side.

Worth noting the scope: this applies only to Unity Catalog pipelines. Hive
metastore pipelines are not affected, so if you have a mix you will see the old
behaviour on some and the new one on others.

One detail that matters operationally: the retained tables can still be
queried, but they cannot be updated until you move them into an active
pipeline. They are not orphaned leftovers, they are queryable data you can
reattach to a pipeline later. That was the point of the change, people deleting
a pipeline and losing the data with it.

For your third question, you do not have to clean up by hand. Set cascade to
true on the delete and you get the old behaviour back:

DELETE /api/2.0/pipelines/{pipeline_id}?cascade=true

If your pipelines get deleted from a CI/CD step rather than the UI, adding that
to the call is probably the whole fix for the manual effort you are describing.

For the tables already orphaned by earlier deletes, just DROP them explicitly.
One warning there, since it is easy to assume otherwise: UNDROP will not save
you on these. Unity Catalog allows UNDROP within 7 days for materialized views
and streaming tables, but only if they were created from an ETL pipeline and
that pipeline still exists. Once the pipeline is gone, dropping the table is a
one-way door. Worth being sure before you run the cleanup.

jlb

View solution in original post

6 REPLIES 6

juanlozadab
Contributor

Hi @surajitDE ,

Yes, this changed, and what you are seeing is the new intended behaviour.

The default flipped on September 25, 2026. From the release notes: when you
delete a Unity Catalog pipeline that uses the default publishing mode, the UI
and the Pipelines DELETE API retain the associated materialized views,
streaming tables and views by default. So nothing is broken on your side.

Worth noting the scope: this applies only to Unity Catalog pipelines. Hive
metastore pipelines are not affected, so if you have a mix you will see the old
behaviour on some and the new one on others.

One detail that matters operationally: the retained tables can still be
queried, but they cannot be updated until you move them into an active
pipeline. They are not orphaned leftovers, they are queryable data you can
reattach to a pipeline later. That was the point of the change, people deleting
a pipeline and losing the data with it.

For your third question, you do not have to clean up by hand. Set cascade to
true on the delete and you get the old behaviour back:

DELETE /api/2.0/pipelines/{pipeline_id}?cascade=true

If your pipelines get deleted from a CI/CD step rather than the UI, adding that
to the call is probably the whole fix for the manual effort you are describing.

For the tables already orphaned by earlier deletes, just DROP them explicitly.
One warning there, since it is easy to assume otherwise: UNDROP will not save
you on these. Unity Catalog allows UNDROP within 7 days for materialized views
and streaming tables, but only if they were created from an ETL pipeline and
that pipeline still exists. Once the pipeline is gone, dropping the table is a
one-way door. Worth being sure before you run the cleanup.

jlb

data_pulse
Contributor

@surajitDE 
Check this link which mentioned in September release that Deleting a Unity catalog pipeline now retains tables by default.

One additional detail: this behaviour was formally rolled out on September 25, 2026. Databricks previously had introduced the cascade option with cascade=true preserving the old deletion behavior, and later changed the default so that Unity Catalog pipeline deletion now retains materialized views, streaming tables, and views.

Also to note that retained objects are marked inactive: they can still be queried, but they will no longer be refreshed until they are moved to an active pipeline. This only applies to Unity Catalog pipelines using the default publishing mode; Hive metastore pipelines are not affected.

For automated cleanup, it would therefore be safer to explicitly use cascade=true rather than relying on the default behavior:
DELETE /api/2.0/pipelines/{pipeline_id}?cascade=true

Reference here

 

anuj_lathi
Databricks Employee
Databricks Employee

Yes, this is expected behavior for Unity Catalog pipelines using the default publishing mode. Deleting the pipeline now retains the materialized views, streaming tables, and views by default, and you can opt back into cascade deletion through the API [1].

What's happening

When you delete a Unity Catalog pipeline in the default publishing mode, the associated materialized views, streaming tables, and views are kept. You can still query them, but they won't be refreshed until you move them into an active pipeline [1]. Deleting from the UI retains the tables by default. This only applies to Unity Catalog pipelines. Hive metastore pipelines aren't affected [1].

To confirm your pipeline is in this mode, either:

  • Check the Summary field in the pipeline settings UI, or
  • Call GET /api/2.0/pipelines/{pipeline_id} and check that effectivePublishingMode is DEFAULT_PUBLISHING_MODE [1].

Getting cascade deletion back

Set cascade to true on the delete request:

DELETE /api/2.0/pipelines/{pipeline_id}?cascade=true

That deletes the pipeline and its tables together [1]. The REST API reference for "Delete a pipeline" has the full request details.

If you're deleting pipelines from CI/CD or scripts, add that parameter to the call. If you use the UI, you'll need to clean up the tables separately, because the UI retains them.

Cleaning up tables manually

For tables that are already orphaned, drop them explicitly:

DROP MATERIALIZED VIEW my_catalog.my_schema.my_mv;
DROP TABLE my_catalog.my_schema.my_streaming_table;

If you drop something by mistake, UNDROP can recover dropped tables within 7 days [1].

A related setting: removing tables from a live pipeline

This is separate from pipeline deletion, but you'll likely run into it. If you remove a table definition from a pipeline's source code, the table is marked inactive on the next update rather than dropped. It stays queryable but is no longer refreshed [1]. To get the older behavior, where the entry is removed from Unity Catalog on the next update, set this pipeline configuration:

"pipelines.dropInactiveTables": "true"

The underlying data is retained for a period, and you can recover it within 7 days by adding the table back to the pipeline definition [1].

Recommendation

  • If you want the old "delete pipeline, delete everything" behavior, use the API with cascade=true.
  • If you delete from the UI, plan a cleanup step to drop the leftover tables. You can list them by schema and script the DROP statements.
  • Consider pipelines.dropInactiveTables if you also want tables removed when you take their definitions out of a pipeline.

References

[1] Use Unity Catalog with pipelines | Databricks on AWS โ€” https://docs.databricks.com/aws/en/ldp/unity-catalog

Anuj Lathi
Solutions Engineer @ Databricks

oliver3681518
New Contributor III

We are using the databricks CLI to create and destroy resources (e.g. in our staging environment). Until this change we were able to do a simple:

databricks bundle destroy

as part of your CI/CD pipeline. This would leave a clean staging environment with all jobs, pipelines and tables gone after a merge into production.

Is there any way to replicate this behaviour using the CLI, or do I have to use the API?

anuj_lathi
Databricks Employee
Databricks Employee

Short answer: I wouldn't rely on databricks bundle destroy to clean up the tables. The retain-by-default behavior applies to Unity Catalog pipelines in the default publishing mode, and the documented way to override it is cascade=true on the delete request [1]. I can't point you to a bundle-level setting that turns that on, so check the release notes for your CLI version before assuming one exists.

Option 1: isolate staging in its own schema and drop it (what I'd do)

Staging is the best case for this approach. Point the bundle's pipelines at a dedicated staging schema, and make teardown two steps:

databricks bundle destroy --auto-approve -t staging
DROP SCHEMA IF EXISTS my_catalog.staging_schema CASCADE;

DROP SCHEMA ... CASCADE removes everything left in the schema, including the retained materialized views and streaming tables. The pipelines and jobs are already gone by then, so nothing is left to refresh them. It also stays correct if tables get added later, so you don't need a list of table names. You can run the SQL from a SQL warehouse, a small cleanup job, or the SQL Statement Execution API via databricks api post, whichever fits your CI.

Dropped tables can be recovered with UNDROP for 7 days [1], which is a useful safety net if the schema name is ever wrong.

Option 2: cascade-delete the pipelines before destroying the bundle

If you want to use the pipeline delete behavior directly, the CLI's databricks api command lets you call the REST endpoint with the cascade parameter [1]:

# get pipeline IDs from the deployed bundle
databricks bundle summary -t staging -o json

# delete each pipeline together with its tables
databricks api delete "/api/2.0/pipelines/<pipeline_id>?cascade=true"

# then clean up the remaining jobs and other resources
databricks bundle destroy --auto-approve -t staging

This has to happen before bundle destroy. Once the pipeline is deleted without cascade, you can't cascade afterwards, and the tables are orphaned. Test how bundle destroy handles pipelines that are already gone before you wire this into CI. It's also more moving parts than Option 1.

Recommendation

Use a dedicated staging schema and add a DROP SCHEMA ... CASCADE step after bundle destroy. Only do this for staging. Never point the same cleanup step at a schema shared with production.

References

[1] Use Unity Catalog with pipelines | Databricks on AWS โ€” https://docs.databricks.com/aws/en/ldp/unity-catalog

Anuj Lathi
Solutions Engineer @ Databricks

oliver3681518
New Contributor III

Thanks for the reply!

One more option I found for declarative pipelines is to set the cascade_on_destroy: true option in the DAB. see https://docs.databricks.com/aws/en/dev-tools/bundles/resources#pipeline 

I set it to false in production and to true in staging, so I get the desired behaviour (deletion of tables) in staging without having to change the Databricks CLI commands in our CI jobs.