<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic Change in DLT Pipeline Deletion Behavior associated Tables Not Being Deleted in Data Engineering</title>
    <link>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/172135#M56580</link>
    <description>&lt;P class=""&gt;Hi Team,&lt;/P&gt;&lt;P&gt;We have noticed a change in the behavior of our Lakeflow pipelines.&lt;/P&gt;&lt;P&gt;Previously, when we deleted a DLT pipeline, the associated resources, such as &lt;STRONG&gt;materialized views and streaming tables&lt;/STRONG&gt;, were also deleted along with the pipeline.&lt;/P&gt;&lt;P&gt;However, recently, when we delete a pipeline, the associated &lt;STRONG&gt;materialized views and streaming tables remain in Unity Catalog&lt;/STRONG&gt; and are not automatically deleted.&lt;/P&gt;&lt;P&gt;Could you please confirm:&lt;/P&gt;&lt;OL&gt;&lt;LI&gt;Has there been any recent update or change in the pipeline deletion behavior?&lt;/LI&gt;&lt;LI&gt;Is this now the expected behavior?&lt;/LI&gt;&lt;LI&gt;Is there any configuration or recommended approach to automatically clean up the associated materialized views and streaming tables when a pipeline is deleted?&lt;/LI&gt;&lt;/OL&gt;&lt;P&gt;This is creating additional manual effort, as we now need to identify and delete the associated resources separately.&lt;/P&gt;</description>
    <pubDate>Wed, 07 Oct 2026 11:45:04 GMT</pubDate>
    <dc:creator>surajitDE</dc:creator>
    <dc:date>2026-10-07T11:45:04Z</dc:date>
    <item>
      <title>Change in DLT Pipeline Deletion Behavior associated Tables Not Being Deleted</title>
      <link>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/172135#M56580</link>
      <description>&lt;P class=""&gt;Hi Team,&lt;/P&gt;&lt;P&gt;We have noticed a change in the behavior of our Lakeflow pipelines.&lt;/P&gt;&lt;P&gt;Previously, when we deleted a DLT pipeline, the associated resources, such as &lt;STRONG&gt;materialized views and streaming tables&lt;/STRONG&gt;, were also deleted along with the pipeline.&lt;/P&gt;&lt;P&gt;However, recently, when we delete a pipeline, the associated &lt;STRONG&gt;materialized views and streaming tables remain in Unity Catalog&lt;/STRONG&gt; and are not automatically deleted.&lt;/P&gt;&lt;P&gt;Could you please confirm:&lt;/P&gt;&lt;OL&gt;&lt;LI&gt;Has there been any recent update or change in the pipeline deletion behavior?&lt;/LI&gt;&lt;LI&gt;Is this now the expected behavior?&lt;/LI&gt;&lt;LI&gt;Is there any configuration or recommended approach to automatically clean up the associated materialized views and streaming tables when a pipeline is deleted?&lt;/LI&gt;&lt;/OL&gt;&lt;P&gt;This is creating additional manual effort, as we now need to identify and delete the associated resources separately.&lt;/P&gt;</description>
      <pubDate>Wed, 07 Oct 2026 11:45:04 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/172135#M56580</guid>
      <dc:creator>surajitDE</dc:creator>
      <dc:date>2026-10-07T11:45:04Z</dc:date>
    </item>
    <item>
      <title>Re: Change in DLT Pipeline Deletion Behavior associated Tables Not Being Deleted</title>
      <link>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/172154#M56584</link>
      <description>&lt;P&gt;Hi &lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/126180"&gt;@surajitDE&lt;/a&gt;&amp;nbsp;,&lt;/P&gt;&lt;P&gt;Yes, this changed, and what you are seeing is the new intended behaviour.&lt;/P&gt;&lt;P&gt;The default flipped on September 25, 2026. From the release notes: when you&lt;BR /&gt;delete a Unity Catalog pipeline that uses the default publishing mode, the UI&lt;BR /&gt;and the Pipelines DELETE API retain the associated materialized views,&lt;BR /&gt;streaming tables and views by default. So nothing is broken on your side.&lt;/P&gt;&lt;P&gt;Worth noting the scope: this applies only to Unity Catalog pipelines. Hive&lt;BR /&gt;metastore pipelines are not affected, so if you have a mix you will see the old&lt;BR /&gt;behaviour on some and the new one on others.&lt;/P&gt;&lt;P&gt;One detail that matters operationally: the retained tables can still be&lt;BR /&gt;queried, but they cannot be updated until you move them into an active&lt;BR /&gt;pipeline. They are not orphaned leftovers, they are queryable data you can&lt;BR /&gt;reattach to a pipeline later. That was the point of the change, people deleting&lt;BR /&gt;a pipeline and losing the data with it.&lt;/P&gt;&lt;P&gt;For your third question, you do not have to clean up by hand. Set cascade to&lt;BR /&gt;true on the delete and you get the old behaviour back:&lt;/P&gt;&lt;P&gt;DELETE /api/2.0/pipelines/{pipeline_id}?cascade=true&lt;/P&gt;&lt;P&gt;If your pipelines get deleted from a CI/CD step rather than the UI, adding that&lt;BR /&gt;to the call is probably the whole fix for the manual effort you are describing.&lt;/P&gt;&lt;P&gt;For the tables already orphaned by earlier deletes, just DROP them explicitly.&lt;BR /&gt;One warning there, since it is easy to assume otherwise: UNDROP will not save&lt;BR /&gt;you on these. Unity Catalog allows UNDROP within 7 days for materialized views&lt;BR /&gt;and streaming tables, but only if they were created from an ETL pipeline and&lt;BR /&gt;that pipeline still exists. Once the pipeline is gone, dropping the table is a&lt;BR /&gt;one-way door. Worth being sure before you run the cleanup.&lt;/P&gt;</description>
      <pubDate>Wed, 07 Oct 2026 13:33:13 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/172154#M56584</guid>
      <dc:creator>juanlozadab</dc:creator>
      <dc:date>2026-10-07T13:33:13Z</dc:date>
    </item>
    <item>
      <title>Re: Change in DLT Pipeline Deletion Behavior associated Tables Not Being Deleted</title>
      <link>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/172157#M56585</link>
      <description>&lt;P&gt;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/126180"&gt;@surajitDE&lt;/a&gt;&amp;nbsp;&lt;BR /&gt;Check this &lt;A href="https://docs.databricks.com/aws/en/release-notes/product/2026/september#deleting-a-unity-catalog-pipeline-now-retains-tables-by-default" target="_self"&gt;link&amp;nbsp;&lt;/A&gt;which mentioned in September release that Deleting a Unity catalog pipeline now retains tables by default.&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;One additional detail:&lt;/STRONG&gt; this behaviour was formally rolled out on September 25, 2026. Databricks previously had &lt;A href="https://docs.databricks.com/aws/en/release-notes/product/2026/april#new-cascade-field-for-deleting-unity-catalog-pipelines-beta" target="_self"&gt;introduced&lt;/A&gt; the cascade option with &lt;STRONG&gt;cascade=true&lt;/STRONG&gt; preserving the old deletion behavior, and later changed the default so that Unity Catalog pipeline deletion now retains materialized views, streaming tables, and views.&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;Also to note that retained objects are marked inactive:&lt;/STRONG&gt; they can still be queried, but they will no longer be refreshed until they are moved to an active pipeline. This only applies to Unity Catalog pipelines using the default publishing mode; Hive metastore pipelines are not affected.&lt;/P&gt;&lt;P&gt;For automated cleanup, it would therefore be safer to explicitly use&lt;STRONG&gt; cascade=true&lt;/STRONG&gt; rather than relying on the default behavior:&lt;BR /&gt;DELETE /api/2.0/pipelines/{pipeline_id}?cascade=true&lt;/P&gt;&lt;P&gt;Reference &lt;A href="https://docs.databricks.com/aws/en/ldp/unity-catalog#delete-a-pipeline" target="_self"&gt;here&lt;/A&gt;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Wed, 07 Oct 2026 13:45:12 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/172157#M56585</guid>
      <dc:creator>data_pulse</dc:creator>
      <dc:date>2026-10-07T13:45:12Z</dc:date>
    </item>
    <item>
      <title>Re: Change in DLT Pipeline Deletion Behavior associated Tables Not Being Deleted</title>
      <link>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/172203#M56596</link>
      <description>&lt;DIV&gt;
&lt;P&gt;Yes, this is expected behavior for Unity Catalog pipelines using the default publishing mode. Deleting the pipeline now retains the materialized views, streaming tables, and views by default, and you can opt back into cascade deletion through the API [1].&lt;/P&gt;
&lt;H2&gt;What's happening&lt;/H2&gt;
&lt;P&gt;When you delete a Unity Catalog pipeline in the default publishing mode, the associated materialized views, streaming tables, and views are kept. You can still query them, but they won't be refreshed until you move them into an active pipeline [1]. Deleting from the UI retains the tables by default. This only applies to Unity Catalog pipelines. Hive metastore pipelines aren't affected [1].&lt;/P&gt;
&lt;P&gt;To confirm your pipeline is in this mode, either:&lt;/P&gt;
&lt;UL&gt;
&lt;LI&gt;Check the &lt;STRONG&gt;Summary&lt;/STRONG&gt; field in the pipeline settings UI, or&lt;/LI&gt;
&lt;LI&gt;Call &lt;CODE&gt;GET /api/2.0/pipelines/{pipeline_id}&lt;/CODE&gt; and check that &lt;CODE&gt;effectivePublishingMode&lt;/CODE&gt; is &lt;CODE&gt;DEFAULT_PUBLISHING_MODE&lt;/CODE&gt; [1].&lt;/LI&gt;
&lt;/UL&gt;
&lt;H2&gt;Getting cascade deletion back&lt;/H2&gt;
&lt;P&gt;Set &lt;CODE&gt;cascade&lt;/CODE&gt; to &lt;CODE&gt;true&lt;/CODE&gt; on the delete request:&lt;/P&gt;
&lt;PRE&gt;&lt;CODE class="language-bash"&gt;DELETE /api/2.0/pipelines/{pipeline_id}?cascade=true
&lt;/CODE&gt;&lt;/PRE&gt;
&lt;P&gt;That deletes the pipeline and its tables together [1]. The REST API reference for "Delete a pipeline" has the full request details.&lt;/P&gt;
&lt;P&gt;If you're deleting pipelines from CI/CD or scripts, add that parameter to the call. If you use the UI, you'll need to clean up the tables separately, because the UI retains them.&lt;/P&gt;
&lt;H2&gt;Cleaning up tables manually&lt;/H2&gt;
&lt;P&gt;For tables that are already orphaned, drop them explicitly:&lt;/P&gt;
&lt;PRE&gt;&lt;CODE class="language-sql"&gt;DROP MATERIALIZED VIEW my_catalog.my_schema.my_mv;
DROP TABLE my_catalog.my_schema.my_streaming_table;
&lt;/CODE&gt;&lt;/PRE&gt;
&lt;P&gt;If you drop something by mistake, &lt;CODE&gt;UNDROP&lt;/CODE&gt; can recover dropped tables within 7 days [1].&lt;/P&gt;
&lt;H2&gt;A related setting: removing tables from a live pipeline&lt;/H2&gt;
&lt;P&gt;This is separate from pipeline deletion, but you'll likely run into it. If you remove a table definition from a pipeline's source code, the table is marked inactive on the next update rather than dropped. It stays queryable but is no longer refreshed [1]. To get the older behavior, where the entry is removed from Unity Catalog on the next update, set this pipeline configuration:&lt;/P&gt;
&lt;PRE&gt;&lt;CODE class="language-json"&gt;"pipelines.dropInactiveTables": "true"
&lt;/CODE&gt;&lt;/PRE&gt;
&lt;P&gt;The underlying data is retained for a period, and you can recover it within 7 days by adding the table back to the pipeline definition [1].&lt;/P&gt;
&lt;H2&gt;Recommendation&lt;/H2&gt;
&lt;UL&gt;
&lt;LI&gt;If you want the old "delete pipeline, delete everything" behavior, use the API with &lt;CODE&gt;cascade=true&lt;/CODE&gt;.&lt;/LI&gt;
&lt;LI&gt;If you delete from the UI, plan a cleanup step to drop the leftover tables. You can list them by schema and script the &lt;CODE&gt;DROP&lt;/CODE&gt; statements.&lt;/LI&gt;
&lt;LI&gt;Consider &lt;CODE&gt;pipelines.dropInactiveTables&lt;/CODE&gt; if you also want tables removed when you take their definitions out of a pipeline.&lt;/LI&gt;
&lt;/UL&gt;
&lt;H2&gt;References&lt;/H2&gt;
&lt;P&gt;[1] Use Unity Catalog with pipelines | Databricks on AWS — &lt;A href="https://docs.databricks.com/aws/en/ldp/unity-catalog" target="_blank"&gt;https://docs.databricks.com/aws/en/ldp/unity-catalog&lt;/A&gt;&lt;/P&gt;
&lt;/DIV&gt;</description>
      <pubDate>Wed, 07 Oct 2026 19:22:57 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/172203#M56596</guid>
      <dc:creator>anuj_lathi</dc:creator>
      <dc:date>2026-10-07T19:22:57Z</dc:date>
    </item>
    <item>
      <title>Re: Change in DLT Pipeline Deletion Behavior associated Tables Not Being Deleted</title>
      <link>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/172401#M56636</link>
      <description>&lt;P&gt;We are using the databricks CLI to create and destroy resources (e.g. in our staging environment). Until this change we were able to do a simple:&lt;/P&gt;&lt;PRE&gt;databricks bundle destroy&lt;/PRE&gt;&lt;P&gt;as part of your CI/CD pipeline. This would leave a clean staging environment with all jobs, pipelines and tables gone after a merge into production.&lt;/P&gt;&lt;P&gt;Is there any way to replicate this behaviour using the CLI, or do I have to use the API?&lt;/P&gt;</description>
      <pubDate>Fri, 09 Oct 2026 09:35:04 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/172401#M56636</guid>
      <dc:creator>oliver3681518</dc:creator>
      <dc:date>2026-10-09T09:35:04Z</dc:date>
    </item>
    <item>
      <title>Re: Change in DLT Pipeline Deletion Behavior associated Tables Not Being Deleted</title>
      <link>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/172406#M56638</link>
      <description>&lt;DIV&gt;
&lt;P&gt;Short answer: I wouldn't rely on &lt;CODE&gt;databricks bundle destroy&lt;/CODE&gt; to clean up the tables. The retain-by-default behavior applies to Unity Catalog pipelines in the default publishing mode, and the documented way to override it is &lt;CODE&gt;cascade=true&lt;/CODE&gt; on the delete request [1]. I can't point you to a bundle-level setting that turns that on, so check the release notes for your CLI version before assuming one exists.&lt;/P&gt;
&lt;H2&gt;Option 1: isolate staging in its own schema and drop it (what I'd do)&lt;/H2&gt;
&lt;P&gt;Staging is the best case for this approach. Point the bundle's pipelines at a dedicated staging schema, and make teardown two steps:&lt;/P&gt;
&lt;PRE&gt;&lt;CODE class="language-bash"&gt;databricks bundle destroy --auto-approve -t staging
&lt;/CODE&gt;&lt;/PRE&gt;
&lt;PRE&gt;&lt;CODE class="language-sql"&gt;DROP SCHEMA IF EXISTS my_catalog.staging_schema CASCADE;
&lt;/CODE&gt;&lt;/PRE&gt;
&lt;P&gt;&lt;CODE&gt;DROP SCHEMA ... CASCADE&lt;/CODE&gt; removes everything left in the schema, including the retained materialized views and streaming tables. The pipelines and jobs are already gone by then, so nothing is left to refresh them. It also stays correct if tables get added later, so you don't need a list of table names. You can run the SQL from a SQL warehouse, a small cleanup job, or the SQL Statement Execution API via &lt;CODE&gt;databricks api post&lt;/CODE&gt;, whichever fits your CI.&lt;/P&gt;
&lt;P&gt;Dropped tables can be recovered with &lt;CODE&gt;UNDROP&lt;/CODE&gt; for 7 days [1], which is a useful safety net if the schema name is ever wrong.&lt;/P&gt;
&lt;H2&gt;Option 2: cascade-delete the pipelines before destroying the bundle&lt;/H2&gt;
&lt;P&gt;If you want to use the pipeline delete behavior directly, the CLI's &lt;CODE&gt;databricks api&lt;/CODE&gt; command lets you call the REST endpoint with the &lt;CODE&gt;cascade&lt;/CODE&gt; parameter [1]:&lt;/P&gt;
&lt;PRE&gt;&lt;CODE class="language-bash"&gt;# get pipeline IDs from the deployed bundle
databricks bundle summary -t staging -o json

# delete each pipeline together with its tables
databricks api delete "/api/2.0/pipelines/&amp;lt;pipeline_id&amp;gt;?cascade=true"

# then clean up the remaining jobs and other resources
databricks bundle destroy --auto-approve -t staging
&lt;/CODE&gt;&lt;/PRE&gt;
&lt;P&gt;This has to happen before &lt;CODE&gt;bundle destroy&lt;/CODE&gt;. Once the pipeline is deleted without cascade, you can't cascade afterwards, and the tables are orphaned. Test how &lt;CODE&gt;bundle destroy&lt;/CODE&gt; handles pipelines that are already gone before you wire this into CI. It's also more moving parts than Option 1.&lt;/P&gt;
&lt;H2&gt;Recommendation&lt;/H2&gt;
&lt;P&gt;Use a dedicated staging schema and add a &lt;CODE&gt;DROP SCHEMA ... CASCADE&lt;/CODE&gt; step after &lt;CODE&gt;bundle destroy&lt;/CODE&gt;. Only do this for staging. Never point the same cleanup step at a schema shared with production.&lt;/P&gt;
&lt;H2&gt;References&lt;/H2&gt;
&lt;P&gt;[1] Use Unity Catalog with pipelines | Databricks on AWS — &lt;A href="https://docs.databricks.com/aws/en/ldp/unity-catalog" target="_blank"&gt;https://docs.databricks.com/aws/en/ldp/unity-catalog&lt;/A&gt;&lt;/P&gt;
&lt;/DIV&gt;</description>
      <pubDate>Fri, 09 Oct 2026 10:23:50 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/172406#M56638</guid>
      <dc:creator>anuj_lathi</dc:creator>
      <dc:date>2026-10-09T10:23:50Z</dc:date>
    </item>
    <item>
      <title>Re: Change in DLT Pipeline Deletion Behavior associated Tables Not Being Deleted</title>
      <link>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/174733#M56901</link>
      <description>&lt;P&gt;Thanks for the reply!&lt;/P&gt;&lt;P&gt;One more option I found for declarative pipelines is to set the &lt;STRONG&gt;cascade_on_destroy: true&lt;/STRONG&gt; option in the DAB. see&amp;nbsp;&lt;A href="https://docs.databricks.com/aws/en/dev-tools/bundles/resources#pipeline" target="_blank" rel="noopener"&gt;https://docs.databricks.com/aws/en/dev-tools/bundles/resources#pipeline&lt;/A&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;I set it to &lt;STRONG&gt;false&amp;nbsp;&lt;/STRONG&gt;in production and to &lt;STRONG&gt;true&lt;/STRONG&gt; in staging, so I get the desired behaviour (deletion of tables) in staging without having to change the Databricks CLI commands in our CI jobs.&lt;/P&gt;</description>
      <pubDate>Tue, 27 Oct 2026 10:16:43 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/change-in-dlt-pipeline-deletion-behavior-associated-tables-not/m-p/174733#M56901</guid>
      <dc:creator>oliver3681518</dc:creator>
      <dc:date>2026-10-27T10:16:43Z</dc:date>
    </item>
  </channel>
</rss>

