https://community.databricks.com/t5/technical-blog/simplifying-production-databricks-pipeline-operations-with/ba-p/119775 <P>Have production Databricks Pipelines on trigger mode and need to change the refresh frequency of some of the Streaming Tables(ST) without a full refresh?</P> <P>This article walks you through an easy, step-by-step guide of how to do exactly that using Databricks Pipeline Move Feature</P> <P>Pipeline Move</P> <UL> <LI>Step 1, Stop the source pipeline if it is running.</LI> <LI>Step 2, Confirm the source pipeline is stopped, then remove the ST definition from the source pipeline’s notebook or file</LI> <LI>Step 3, Reassign the ST from the source pipeline to the destination pipeline:</LI> </UL> <P>ALTER STREAMING TABLEmy_catalog.my_schema.my_table SET TBLPROPERTIES ("pipelines.pipelineId"="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx");</P> <UL> <LI>Step 4, Add the ST’s definition to the destination pipeline’s notebook or file.</LI> </UL> <P>This will allow the destination pipeline to update the ST instead of the source pipeline.</P> <P>Just like that, now you have successfully moved ST from source pipeline line to target pipeline without any full refresh in 4 simple steps!</P> <P>To use the Databricks pipeline move feature, the following requirements must be met:</P> <UL> <LI>Both the source and destination pipelines must be Unity Catalog (UC) pipelines.</LI> <LI>Both pipelines must reside within the same Databricks workspace.</LI> <LI>The user performing the operation must own both the source and destination pipelines.</LI> <LI>The destination pipeline must use the default publishing mode, which enables publishing tables to multiple catalogs and schemas.</LI> </UL> <P><STRONG>Tips and Tricks</STRONG></P> <UL> <LI>To move temporary ST from source pipeline to destination pipeline, first set temporary=False, then perform Steps 1 to 4, for example: <a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/97035">@Dlt</a>.table(name='my_temp_st', temporary=False)</LI> <LI>To solve the “Table already exist” error during the move, check if you follow the order of the steps 1 to 4: make sure removing ST definition from source pipeline notebook or file BEFORE switching pipeline id to destination pipeline using Alter Streaming Table</LI> </UL> <P>Databricks pipeline move feature is also useful in scenarios where you may want to split a large pipeline into smaller ones or merge several pipelines together.</P> <P>Happy Pipelining!</P> <P>&nbsp;</P> Wed, 28 May 2025 14:48:29 GMT Shu_Li 2025-05-28T14:48:29Z https://community.databricks.com/t5/technical-blog/simplifying-production-databricks-pipeline-operations-with/ba-p/119775 <P>Have production Databricks pipelines hydrating significant sized LakeHouse but need to either split or merge pipelines to establish new ingestion boundaries? Or change the ingestion frequencies of some Streaming Tables or Materialized Views in the same triggered pipeline? We got you! In this article, I will show you how to do this in 4 easy steps using Databricks Pipeline Move feature without any full refresh of the pipelines!</P> Wed, 28 May 2025 14:48:29 GMT https://community.databricks.com/t5/technical-blog/simplifying-production-databricks-pipeline-operations-with/ba-p/119775 Shu_Li 2025-05-28T14:48:29Z https://community.databricks.com/t5/technical-blog/simplifying-production-databricks-pipeline-operations-with/bc-p/120839#M624 <P><span class="lia-unicode-emoji" title=":sparkler:">🎇</span>Update: Now you can move pipelines across different workspaces<span class="lia-unicode-emoji" title=":party_popper:">🎉</span></P> <P>Happy pipelining<span class="lia-unicode-emoji" title=":rocket:">🚀</span></P> Tue, 03 Jun 2025 17:04:36 GMT https://community.databricks.com/t5/technical-blog/simplifying-production-databricks-pipeline-operations-with/bc-p/120839#M624 Shu_Li 2025-06-03T17:04:36Z https://community.databricks.com/t5/technical-blog/simplifying-production-databricks-pipeline-operations-with/bc-p/120848#M625 <P><span class="lia-unicode-emoji" title=":backhand_index_pointing_right:">👉</span>check out more detail in our public documentation:&nbsp;<A class="c-link" href="https://docs.databricks.com/aws/en/dlt/move-table" target="_blank" rel="noopener noreferrer" data-stringify-link="https://docs.databricks.com/aws/en/dlt/move-table" data-sk="tooltip_parent">https://docs.databricks.com/aws/en/dlt/move-table</A><SPAN>&nbsp;</SPAN></P> Tue, 03 Jun 2025 18:32:48 GMT https://community.databricks.com/t5/technical-blog/simplifying-production-databricks-pipeline-operations-with/bc-p/120848#M625 Shu_Li 2025-06-03T18:32:48Z