Hi @ajay_wavicle,
This is a common scenario when migrating Unity Catalog managed tables across workspaces, especially on Azure where the managed storage account has a deny assignment that blocks direct file-level access.
You are correct that DEEP CLONE does not preserve the full Delta version history. A cloned table starts with a fresh transaction log, so time travel on the clone uses new version numbers independent of the source.
Here are the approaches to consider, depending on what you mean by "retaining history":
OPTION 1: DELTA SHARING WITH HISTORY (RECOMMENDED FOR TIME TRAVEL ACCESS)
If both workspaces are on the same Databricks account (or even across accounts), Delta Sharing lets you share managed UC tables with full history enabled. Recipients can then run time travel queries against the shared data.
On the source workspace (provider side):
CREATE SHARE my_migration_share;
ALTER SHARE my_migration_share ADD TABLE catalog.schema.my_table WITH HISTORY;
CREATE RECIPIENT target_workspace;
GRANT SELECT ON SHARE my_migration_share TO RECIPIENT target_workspace;
On the target workspace (recipient side), once the share is configured, you can create a catalog from the share and query it. If you need a fully independent local copy, you can then run CTAS or DEEP CLONE from the shared catalog into a local managed table. The shared version will retain the original history for time travel while the local copy will have its own history going forward.
This requires Databricks Runtime 12.2 LTS or above for history sharing. On DBR 16.2+, history sharing is enabled by default.
Documentation: https://docs.databricks.com/en/delta-sharing/create-share.html
OPTION 2: DEEP CLONE (DATA + METADATA, BUT FRESH HISTORY)
If you only need the current data and metadata (schema, partitioning, constraints) but can accept a fresh version history, DEEP CLONE is the simplest approach. Since both workspaces share the same Unity Catalog metastore (or you can register the target workspace to the same metastore), you can clone directly:
CREATE TABLE target_catalog.target_schema.my_table
DEEP CLONE source_catalog.source_schema.my_table;
This copies all data files and metadata. The clone will have version 0 as its starting point. Stream metadata and COPY INTO metadata are also preserved, so interrupted pipelines can resume.
Documentation: https://docs.databricks.com/en/delta/clone.html
OPTION 3: SHARED METASTORE (NO COPY NEEDED)
If both workspaces can be registered to the same Unity Catalog metastore, no data migration is needed at all. Unity Catalog managed tables are accessible from any workspace attached to the same metastore. You simply assign both workspaces to the metastore and grant the appropriate permissions.
This is the ideal approach when both workspaces are in the same region and account.
WHY DIRECT FILE COPY DOES NOT WORK FOR UC MANAGED TABLES
As you discovered, Unity Catalog managed storage on Azure uses a deny assignment to prevent direct access to the underlying storage account. This is by design: all access to managed tables must go through Unity Catalog's governance layer. Using dbutils.fs.cp against the managed storage path will fail because of this restriction.
For external tables, you have direct storage access, which is why file copy works there. For managed tables, you need to use one of the approaches above.
SUMMARY
- Need time travel on original history: Use Delta Sharing with history enabled
- Need a full independent copy: Use DEEP CLONE (accepts fresh history)
- Same account, same region: Consider attaching both workspaces to the same metastore (no copy needed)
- Direct file copy from managed storage: Not supported due to deny assignment on Azure
* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.
If this answer resolves your question, could you mark it as "Accept as Solution"? That helps other users quickly find the correct fix.
