DLT pipeline cloning to another workspace.

[yanchr]

I know about the clone API:
<databricks-instance>/api/2.0/pipelines/<pipeline-id>/clone

but it only works within the same workspace.

Is it possible to achieve the same result manually?

As I understand:

• I can't choose where checkpoints are migrated to in Unity Catalog.
• By design, I can't find where a given pipeline's checkpoints are stored in UC.

Given that, what's the actual solution here? Is rewriting the DLT pipeline as raw Structured Streaming (with an explicit checkpointLocation) the only way to do this manually?

[balajij8]

The right way to handle this is by using DABs. Instead of manually moving state or dropping down to raw Structured Streaming, you can use a bundle to deploy the pipeline configuration and underlying code to the target workspace. When the pipeline runs in the new workspace, DLT handles the state automatically by creating new checkpoints in the target workspace's UC managed storage. This is strictly by design. Depending on the source, the new pipeline will either start fresh, require a full refresh to catch up and then seamlessly resume from the new checkpoint location going forward. More details here

You can achieve this manually if DABs are not an option for your current setup. You can export the pipeline configuration JSON from the source workspace, migrate the transformation files, and deploy a new pipeline in the target workspace using the same configuration. The new pipeline gets fresh checkpoints automatically.

To isolate the data side of things, if both workspaces share the same Unity Catalog metastore, your output tables are most likely accessible across both environments. You need to recreate the pipeline definition in the target workspace pointing to those same UC catalogs, and DLT will spin up the necessary fresh checkpoints for the new pipeline ID under the hood.

[yanchr]

The problem is my source pipeline uses hive metastore and I'm wondering if its possible to migrate data in table too and save old checkpoints

[balajij8]

You can follow below

• You can check copying delta tables using DEEP CLONE. It preserves history. You can recreate pipeline using DABs or manual configurations. You can accept fresh checkpoints in target workspace or configure streaming sources to resume from recent data by using modifiedAfter to pick up only the new data after the deep clone was done.
• You can also upgrade source pipeline to Unity Catalog and use DABs for cross-workspace deployment. No data copy is required if its shared meta store. You can use Delta Sharing or DEEP CLONE otherwise.

[henry_collins]

From what I've seen, there isn't a supported manual way to migrate a DLT pipeline with its checkpoints across workspaces. The checkpoint location is managed by the platform, so you don't get control over moving or reusing it. If preserving streaming state across workspaces is a hard requirement, then using raw Structured Streaming with your own checkpointLocation is generally the more flexible approach.