โ10-07-2026 04:45 AM
Hi Team,
We have noticed a change in the behavior of our Lakeflow pipelines.
Previously, when we deleted a DLT pipeline, the associated resources, such as materialized views and streaming tables, were also deleted along with the pipeline.
However, recently, when we delete a pipeline, the associated materialized views and streaming tables remain in Unity Catalog and are not automatically deleted.
Could you please confirm:
This is creating additional manual effort, as we now need to identify and delete the associated resources separately.
โ10-07-2026 06:33 AM
Hi @surajitDE ,
Yes, this changed, and what you are seeing is the new intended behaviour.
The default flipped on September 25, 2026. From the release notes: when you
delete a Unity Catalog pipeline that uses the default publishing mode, the UI
and the Pipelines DELETE API retain the associated materialized views,
streaming tables and views by default. So nothing is broken on your side.
Worth noting the scope: this applies only to Unity Catalog pipelines. Hive
metastore pipelines are not affected, so if you have a mix you will see the old
behaviour on some and the new one on others.
One detail that matters operationally: the retained tables can still be
queried, but they cannot be updated until you move them into an active
pipeline. They are not orphaned leftovers, they are queryable data you can
reattach to a pipeline later. That was the point of the change, people deleting
a pipeline and losing the data with it.
For your third question, you do not have to clean up by hand. Set cascade to
true on the delete and you get the old behaviour back:
DELETE /api/2.0/pipelines/{pipeline_id}?cascade=true
If your pipelines get deleted from a CI/CD step rather than the UI, adding that
to the call is probably the whole fix for the manual effort you are describing.
For the tables already orphaned by earlier deletes, just DROP them explicitly.
One warning there, since it is easy to assume otherwise: UNDROP will not save
you on these. Unity Catalog allows UNDROP within 7 days for materialized views
and streaming tables, but only if they were created from an ETL pipeline and
that pipeline still exists. Once the pipeline is gone, dropping the table is a
one-way door. Worth being sure before you run the cleanup.
โ10-07-2026 06:33 AM
Hi @surajitDE ,
Yes, this changed, and what you are seeing is the new intended behaviour.
The default flipped on September 25, 2026. From the release notes: when you
delete a Unity Catalog pipeline that uses the default publishing mode, the UI
and the Pipelines DELETE API retain the associated materialized views,
streaming tables and views by default. So nothing is broken on your side.
Worth noting the scope: this applies only to Unity Catalog pipelines. Hive
metastore pipelines are not affected, so if you have a mix you will see the old
behaviour on some and the new one on others.
One detail that matters operationally: the retained tables can still be
queried, but they cannot be updated until you move them into an active
pipeline. They are not orphaned leftovers, they are queryable data you can
reattach to a pipeline later. That was the point of the change, people deleting
a pipeline and losing the data with it.
For your third question, you do not have to clean up by hand. Set cascade to
true on the delete and you get the old behaviour back:
DELETE /api/2.0/pipelines/{pipeline_id}?cascade=true
If your pipelines get deleted from a CI/CD step rather than the UI, adding that
to the call is probably the whole fix for the manual effort you are describing.
For the tables already orphaned by earlier deletes, just DROP them explicitly.
One warning there, since it is easy to assume otherwise: UNDROP will not save
you on these. Unity Catalog allows UNDROP within 7 days for materialized views
and streaming tables, but only if they were created from an ETL pipeline and
that pipeline still exists. Once the pipeline is gone, dropping the table is a
one-way door. Worth being sure before you run the cleanup.
โ10-07-2026 06:45 AM
@surajitDE
Check this link which mentioned in September release that Deleting a Unity catalog pipeline now retains tables by default.
One additional detail: this behaviour was formally rolled out on September 25, 2026. Databricks previously had introduced the cascade option with cascade=true preserving the old deletion behavior, and later changed the default so that Unity Catalog pipeline deletion now retains materialized views, streaming tables, and views.
Also to note that retained objects are marked inactive: they can still be queried, but they will no longer be refreshed until they are moved to an active pipeline. This only applies to Unity Catalog pipelines using the default publishing mode; Hive metastore pipelines are not affected.
For automated cleanup, it would therefore be safer to explicitly use cascade=true rather than relying on the default behavior:
DELETE /api/2.0/pipelines/{pipeline_id}?cascade=true
Reference here
โ10-07-2026 12:22 PM
Yes, this is expected behavior for Unity Catalog pipelines using the default publishing mode. Deleting the pipeline now retains the materialized views, streaming tables, and views by default, and you can opt back into cascade deletion through the API [1].
When you delete a Unity Catalog pipeline in the default publishing mode, the associated materialized views, streaming tables, and views are kept. You can still query them, but they won't be refreshed until you move them into an active pipeline [1]. Deleting from the UI retains the tables by default. This only applies to Unity Catalog pipelines. Hive metastore pipelines aren't affected [1].
To confirm your pipeline is in this mode, either:
GET /api/2.0/pipelines/{pipeline_id} and check that effectivePublishingMode is DEFAULT_PUBLISHING_MODE [1].Set cascade to true on the delete request:
DELETE /api/2.0/pipelines/{pipeline_id}?cascade=true
That deletes the pipeline and its tables together [1]. The REST API reference for "Delete a pipeline" has the full request details.
If you're deleting pipelines from CI/CD or scripts, add that parameter to the call. If you use the UI, you'll need to clean up the tables separately, because the UI retains them.
For tables that are already orphaned, drop them explicitly:
DROP MATERIALIZED VIEW my_catalog.my_schema.my_mv;
DROP TABLE my_catalog.my_schema.my_streaming_table;
If you drop something by mistake, UNDROP can recover dropped tables within 7 days [1].
This is separate from pipeline deletion, but you'll likely run into it. If you remove a table definition from a pipeline's source code, the table is marked inactive on the next update rather than dropped. It stays queryable but is no longer refreshed [1]. To get the older behavior, where the entry is removed from Unity Catalog on the next update, set this pipeline configuration:
"pipelines.dropInactiveTables": "true"
The underlying data is retained for a period, and you can recover it within 7 days by adding the table back to the pipeline definition [1].
cascade=true.DROP statements.pipelines.dropInactiveTables if you also want tables removed when you take their definitions out of a pipeline.[1] Use Unity Catalog with pipelines | Databricks on AWS โ https://docs.databricks.com/aws/en/ldp/unity-catalog
โ10-09-2026 02:35 AM
We are using the databricks CLI to create and destroy resources (e.g. in our staging environment). Until this change we were able to do a simple:
databricks bundle destroy
as part of your CI/CD pipeline. This would leave a clean staging environment with all jobs, pipelines and tables gone after a merge into production.
Is there any way to replicate this behaviour using the CLI, or do I have to use the API?
โ10-09-2026 03:23 AM
Short answer: I wouldn't rely on databricks bundle destroy to clean up the tables. The retain-by-default behavior applies to Unity Catalog pipelines in the default publishing mode, and the documented way to override it is cascade=true on the delete request [1]. I can't point you to a bundle-level setting that turns that on, so check the release notes for your CLI version before assuming one exists.
Staging is the best case for this approach. Point the bundle's pipelines at a dedicated staging schema, and make teardown two steps:
databricks bundle destroy --auto-approve -t staging
DROP SCHEMA IF EXISTS my_catalog.staging_schema CASCADE;
DROP SCHEMA ... CASCADE removes everything left in the schema, including the retained materialized views and streaming tables. The pipelines and jobs are already gone by then, so nothing is left to refresh them. It also stays correct if tables get added later, so you don't need a list of table names. You can run the SQL from a SQL warehouse, a small cleanup job, or the SQL Statement Execution API via databricks api post, whichever fits your CI.
Dropped tables can be recovered with UNDROP for 7 days [1], which is a useful safety net if the schema name is ever wrong.
If you want to use the pipeline delete behavior directly, the CLI's databricks api command lets you call the REST endpoint with the cascade parameter [1]:
# get pipeline IDs from the deployed bundle
databricks bundle summary -t staging -o json
# delete each pipeline together with its tables
databricks api delete "/api/2.0/pipelines/<pipeline_id>?cascade=true"
# then clean up the remaining jobs and other resources
databricks bundle destroy --auto-approve -t staging
This has to happen before bundle destroy. Once the pipeline is deleted without cascade, you can't cascade afterwards, and the tables are orphaned. Test how bundle destroy handles pipelines that are already gone before you wire this into CI. It's also more moving parts than Option 1.
Use a dedicated staging schema and add a DROP SCHEMA ... CASCADE step after bundle destroy. Only do this for staging. Never point the same cleanup step at a schema shared with production.
[1] Use Unity Catalog with pipelines | Databricks on AWS โ https://docs.databricks.com/aws/en/ldp/unity-catalog
2 weeks ago
Thanks for the reply!
One more option I found for declarative pipelines is to set the cascade_on_destroy: true option in the DAB. see https://docs.databricks.com/aws/en/dev-tools/bundles/resources#pipeline
I set it to false in production and to true in staging, so I get the desired behaviour (deletion of tables) in staging without having to change the Databricks CLI commands in our CI jobs.