cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

how to change delta table to managed iceberg table by writing it metadata & using same parquet data

ajay_wavicle
Visitor

i need to change delta table to managed iceberg table by writing it metadata and using same parquet data and not rewriting the table efficiently. Dont want to use Delta uniform format

2 ACCEPTED SOLUTIONS

Accepted Solutions

Louis_Frolio
Databricks Employee
Databricks Employee

Greetings @ajay_wavicle , I did some digging as this is a "not so common" request.  With that said, here is what I found.

What you’re aiming for is very specific: you want to end up with a Unity Catalog managed Apache Iceberg table, reusing the exact same Parquet files that back an existing Delta table, writing only new metadata (no data rewrite), and explicitly without using Delta UniForm.

Let’s walk through what’s actually supported today, what isn’t, and what your real options are.

What’s supported today (and what isn’t)

Today, the only documented, metadata-only, no-rewrite way to expose an existing Delta table’s Parquet files to Iceberg readers is to enable Iceberg compatibility via Delta UniForm. UniForm generates Iceberg metadata alongside the Delta transaction log and allows Iceberg engines to read the table without rewriting any data.

Managed Iceberg tables in Unity Catalog are a different thing. They are first-class Iceberg objects: writable via the Iceberg REST catalog and fully integrated with Databricks platform features. However, creating a managed Iceberg table is typically done by creating a brand-new Iceberg table (for example via CTAS), which rewrites data.

Internal guidance today is to avoid migrating existing Delta tables to managed Iceberg unless you have a concrete need for Iceberg writes from external engines. Databricks’ longer-term direction is format convergence, and the recommended interoperability path from Delta remains UniForm — which you’ve already ruled out.

So there’s no magic switch hiding somewhere.

Practical options

Option A — Minimal friction, no rewrite: enable UniForm on the Delta table

This keeps the table as a Delta table managed by Unity Catalog, but generates and maintains Iceberg metadata so Iceberg clients can read it. No Parquet files are rewritten.

This requires a UC-registered table and column mapping enabled. Writes require DBR 14.3+. You enable it with ALTER TABLE and can verify via SHOW TBLPROPERTIES or DESCRIBE EXTENDED.

Example:

ALTER TABLE catalog.schema.delta_tbl SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

If you need to force a metadata sync:

MSCK REPAIR TABLE catalog.schema.delta_tbl SYNC METADATA;

 

Option B — True managed Iceberg object: create a new managed Iceberg table

If you need a first-class managed Iceberg table (for native Iceberg writes via REST catalog and full platform integration), the supported path is to create a new Iceberg table using CTAS or INSERT INTO. This rewrites data.

Example:

CREATE TABLE catalog.schema.ice_tbl
USING ICEBERG
AS SELECT * FROM catalog.schema.delta_tbl;

Option C — Two-step “no rewrite” path via foreign Iceberg (advanced, caveats apply)

If you absolutely must avoid a rewrite and must end with a managed Iceberg table, there is a possible but non-trivial route:

First, use open-source Iceberg tooling to create or register a foreign Iceberg table that points at the existing Parquet files. This is a metadata-only operation, but it requires constructing valid Iceberg metadata and manifests outside of Unity Catalog (for example in Glue or HMS). This is not a productized Databricks workflow.

Second, in Unity Catalog, convert that foreign Iceberg table into a managed Iceberg table. UC does support foreign-Iceberg → managed-Iceberg conversions, and this step does not rewrite data.

This works, but it’s operationally complex, requires external catalog plumbing, and is not the “in-place flip” most teams expect. It’s best reserved for cases where external Iceberg writes are mandatory and a rewrite is truly impossible.

Important limitations and gotchas

If you later enable Liquid Clustering on a managed Iceberg table, deletion vectors and row tracking must be disabled first. Otherwise you’ll hit errors due to Iceberg v2 concurrency limitations.

If you do consider UniForm at any point, be aware that Delta and Iceberg versions do not advance in lockstep. Databricks tracks converted_delta_version and converted_delta_timestamp to relate them, and you may need to explicitly sync metadata in some workflows.

Bottom line / recommendation

Given your constraints — no rewrite and no UniForm — there is currently no documented, in-place, metadata-only command in Databricks that converts a Delta table directly into a Unity Catalog managed Iceberg table.

Your supported choices today are:

• Enable UniForm for no-rewrite Iceberg read interoperability

• Accept a one-time CTAS rewrite to create a managed Iceberg table

• Or pursue the foreign Iceberg → managed Iceberg two-step using external tooling if you must end with managed Iceberg without rewriting Parquet data

That’s the honest state of the world right now.

Hope this helps you address your task.

Cheers, Louis.

View solution in original post

3 REPLIES 3

Louis_Frolio
Databricks Employee
Databricks Employee

Greetings @ajay_wavicle , I did some digging as this is a "not so common" request.  With that said, here is what I found.

What you’re aiming for is very specific: you want to end up with a Unity Catalog managed Apache Iceberg table, reusing the exact same Parquet files that back an existing Delta table, writing only new metadata (no data rewrite), and explicitly without using Delta UniForm.

Let’s walk through what’s actually supported today, what isn’t, and what your real options are.

What’s supported today (and what isn’t)

Today, the only documented, metadata-only, no-rewrite way to expose an existing Delta table’s Parquet files to Iceberg readers is to enable Iceberg compatibility via Delta UniForm. UniForm generates Iceberg metadata alongside the Delta transaction log and allows Iceberg engines to read the table without rewriting any data.

Managed Iceberg tables in Unity Catalog are a different thing. They are first-class Iceberg objects: writable via the Iceberg REST catalog and fully integrated with Databricks platform features. However, creating a managed Iceberg table is typically done by creating a brand-new Iceberg table (for example via CTAS), which rewrites data.

Internal guidance today is to avoid migrating existing Delta tables to managed Iceberg unless you have a concrete need for Iceberg writes from external engines. Databricks’ longer-term direction is format convergence, and the recommended interoperability path from Delta remains UniForm — which you’ve already ruled out.

So there’s no magic switch hiding somewhere.

Practical options

Option A — Minimal friction, no rewrite: enable UniForm on the Delta table

This keeps the table as a Delta table managed by Unity Catalog, but generates and maintains Iceberg metadata so Iceberg clients can read it. No Parquet files are rewritten.

This requires a UC-registered table and column mapping enabled. Writes require DBR 14.3+. You enable it with ALTER TABLE and can verify via SHOW TBLPROPERTIES or DESCRIBE EXTENDED.

Example:

ALTER TABLE catalog.schema.delta_tbl SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

If you need to force a metadata sync:

MSCK REPAIR TABLE catalog.schema.delta_tbl SYNC METADATA;

 

Option B — True managed Iceberg object: create a new managed Iceberg table

If you need a first-class managed Iceberg table (for native Iceberg writes via REST catalog and full platform integration), the supported path is to create a new Iceberg table using CTAS or INSERT INTO. This rewrites data.

Example:

CREATE TABLE catalog.schema.ice_tbl
USING ICEBERG
AS SELECT * FROM catalog.schema.delta_tbl;

Option C — Two-step “no rewrite” path via foreign Iceberg (advanced, caveats apply)

If you absolutely must avoid a rewrite and must end with a managed Iceberg table, there is a possible but non-trivial route:

First, use open-source Iceberg tooling to create or register a foreign Iceberg table that points at the existing Parquet files. This is a metadata-only operation, but it requires constructing valid Iceberg metadata and manifests outside of Unity Catalog (for example in Glue or HMS). This is not a productized Databricks workflow.

Second, in Unity Catalog, convert that foreign Iceberg table into a managed Iceberg table. UC does support foreign-Iceberg → managed-Iceberg conversions, and this step does not rewrite data.

This works, but it’s operationally complex, requires external catalog plumbing, and is not the “in-place flip” most teams expect. It’s best reserved for cases where external Iceberg writes are mandatory and a rewrite is truly impossible.

Important limitations and gotchas

If you later enable Liquid Clustering on a managed Iceberg table, deletion vectors and row tracking must be disabled first. Otherwise you’ll hit errors due to Iceberg v2 concurrency limitations.

If you do consider UniForm at any point, be aware that Delta and Iceberg versions do not advance in lockstep. Databricks tracks converted_delta_version and converted_delta_timestamp to relate them, and you may need to explicitly sync metadata in some workflows.

Bottom line / recommendation

Given your constraints — no rewrite and no UniForm — there is currently no documented, in-place, metadata-only command in Databricks that converts a Delta table directly into a Unity Catalog managed Iceberg table.

Your supported choices today are:

• Enable UniForm for no-rewrite Iceberg read interoperability

• Accept a one-time CTAS rewrite to create a managed Iceberg table

• Or pursue the foreign Iceberg → managed Iceberg two-step using external tooling if you must end with managed Iceberg without rewriting Parquet data

That’s the honest state of the world right now.

Hope this helps you address your task.

Cheers, Louis.

thanks Louis!

@Louis_Frolio , Can you help me in moving managed uc delta table from one storage account in databricks along with its version time travel history to another cloud databricks workspace? I see deny assignment to databricks associated storage account. I want recreate the tables and other as it is.