Unity Catalog Managed Iceberg Tables: What They Actually Are, and What Tripped Me Up

ozaaditya — Wed, 12 Aug 2026 04:12:58 GMT

I've been sitting on this one for a few weeks because I wanted to actually use the feature before writing about it, not just paraphrase the release notes. So this isn't a "here's the announcement" post. It's more of a "here's what I learned after turning it on and running into a couple of walls" post.

If you've been anywhere near the Lakehouse-vs-Iceberg conversation over the last year, you know the format wars have basically ended in a truce. Everybody wants to read everybody's tables. Databricks going all-in on managed Iceberg tables inside Unity Catalog is a big part of that, and I think it's worth understanding what's really happening under the hood before you start creating them everywhere.

The one-line version

You can now create a native Apache Iceberg table that Unity Catalog owns and manages end to end storage, maintenance, optimization, access control, the whole thing. Not a Delta table wearing an Iceberg costume (that's the older Uniform / "Iceberg reads on Delta" approach, which still exists and is still useful). An actual Iceberg table, written in the Iceberg spec, that external Iceberg engines can read and write through the Iceberg REST Catalog.

That last part is the bit that changed how I think about it. It's not just "Databricks can now speak Iceberg." It's "your Trino or Spark or Flink job can write into a table that Unity Catalog governs." The catalog is the boundary, not the compute engine.

At the time of writing this is in Public Preview, and you need Databricks Runtime 16.4 LTS or above. Keep that version number in mind, because I'll come back to it. A couple of the nicer features need something even newer.

Creating one is almost boringly simple

Here's the whole thing:

CREATE TABLE my_catalog.my_schema.events (
  event_id   BIGINT,
  user_id    BIGINT,
  event_type STRING,
  event_ts   TIMESTAMP
)
USING iceberg;

That USING iceberg clause is the entire difference. Leave it off and you get a managed Delta table (still the default). Put it on and Unity Catalog creates a managed Iceberg table instead. Same three-level namespace, same governance model, same GRANT statements you already know.

Whatever you throw at it (SQL, PySpark, saveAsTable), you don't manage a location, you don't hand it a path, you don't think about where the files live. That's the "managed" promise, and honestly it holds up.

Now the stuff nobody tells you until you hit it

This is the part I actually wanted to write. The docs are correct but they bury a few things that will absolutely stop you cold on first attempt.

You need serverless enabled. Unity Catalog uses serverless compute behind the scenes to maintain the Iceberg metadata for these tables. If your workspace doesn't have serverless, you can't create managed Iceberg tables. Full stop. And if your storage account sits behind a firewall, serverless needs a network path to it, or the metadata maintenance silently has nowhere to go. Worth checking your serverless networking setup before you promise anyone a demo.

Predictive Optimization has to be on. This one got me. A managed Iceberg table can only be created if predictive optimization is enabled for table maintenance. It makes sense once you think about it, since the whole model assumes Databricks is handling compaction and snapshot expiration for you, but the error message wasn't obvious to me in the moment. If your create statement fails for a reason you can't explain, check PO first.

Forget PARTITIONED BY. For managed Iceberg tables, classic partitioning isn't supported through Databricks SQL. You use liquid clustering instead:

CREATE TABLE my_catalog.my_schema.events (
  event_id   BIGINT,
  user_id    BIGINT,
  event_type STRING,
  event_ts   TIMESTAMP
)
USING iceberg
CLUSTER BY (event_ts);

If you're coming from a Hive or older Iceberg background, this is a mindset shift. There's no days(event_ts) transform, no bucketing. Those expression-based partition transforms just aren't there for managed tables. And a small gotcha: when you use CLUSTER BY on an Iceberg table you have to explicitly disable deletion vectors and row IDs. Not intuitive, but that's the rule today.

Partition evolution is external-only. You can do partition evolution (ADD PARTITION FIELD, DROP PARTITION FIELD, REPLACE PARTITION FIELD), but only from an external Iceberg engine going through the REST Catalog, not from Databricks SQL. So it exists, it just doesn't live where you'd expect it to.

The interoperability story is the real selling point

Here's where I think the feature earns its keep. Every managed Iceberg table in Unity Catalog is reachable over the Iceberg REST Catalog API. That means engines like Apache Spark, Flink, Trino, DuckDB, Daft, and Kafka can connect straight to Unity Catalog and work with the table, read and write, without Databricks compute in the loop at all.

The mechanism that makes this safe is credential vending: the REST Catalog hands the external engine short-lived, scoped credentials to reach the underlying storage, so you're not sprinkling long-lived keys across your infrastructure. Governance stays in Unity Catalog even when the compute doesn't.

One practical note from the trenches: use Iceberg client 1.9.2 or newer. Older clients technically connect but you'll chase weird behavior. Pin the version and save yourself the afternoon I lost.

(Small warning worth repeating: credential vending isn't supported on workspaces using default storage. If that's your setup, external write access won't behave the way you expect.)

Versions matter more than usual here

Databricks supports Iceberg spec v1, v2, and v3, and the differences aren't cosmetic:

On v2, position deletes and equality deletes aren't supported. If you want efficient row-level deletes, you want v3.
v3 brings deletion vectors, the VARIANT type for semi-structured data, and row lineage. To create a v3 table you just set the property:

CREATE OR REPLACE TABLE my_catalog.my_schema.events (event_id BIGINT)
USING iceberg
TBLPROPERTIES ('format-version' = 3);

But here's the version thing I flagged earlier: some of the shinier v3 capabilities (geospatial types, for example) need DBR 18 LTS or above, not just 16.4. So "supported" and "supported on my cluster" are two different questions. Check both.

Things it can't do yet (as of this preview)

I'd rather you hear these from a blog than discover them in a sprint:

Parquet only. That's the only file format managed Iceberg tables support right now.
No AI Search on managed Iceberg tables.
Compression is Zstd, and you can't change the codec.
A pile of Delta niceties simply don't apply because they're not in the Iceberg spec. Generated columns, constraints, and collation are all off the table.
A few data types aren't supported: UUID, Fixed(L), TIME, and nested STRUCT with required fields.

None of these were dealbreakers for me, but the data-type list in particular is the kind of thing that bites you three tables into a migration.

A couple of nice extras

If you want a clean, independent copy of a table, DEEP CLONE works and produces a brand-new managed Iceberg table with its own data and metadata:

CREATE TABLE my_catalog.my_schema.events_copy
DEEP CLONE my_catalog.my_schema.events;

There are also managed Iceberg materialized views that external Iceberg readers can consume (you create them with USING ICEBERG and then run REPAIR TABLE ... SYNC METADATA). That one's in preview and gated (you have to ping your Databricks account team to switch it on), so I'm mentioning it more as a "this is coming" than a "go use it today."

So should you migrate everything tomorrow?

No, and I don't think that's the point. If your world is entirely inside Databricks and you're happy with Delta, managed Iceberg tables don't magically make your queries faster. You already had liquid clustering and predictive optimization on Delta.

Where this genuinely matters is when you have other engines in the mix. A Trino cluster your analysts love, a Flink pipeline, a partner who standardized on Iceberg, a multi-engine architecture you're trying to keep governed from one place. That's the scenario where "Unity Catalog governs a real Iceberg table that anything can read and write" stops being a spec detail and starts being the reason you sleep at night.

I've moved a couple of shared, cross-team tables over and left everything else on Delta for now. That feels like the right instinct at the preview stage.

If you've been running these in your own workspace, I'd love to hear what broke for you, especially anyone doing writes from external engines at any real volume. That's the corner I've poked at least, and it's where I suspect the interesting war stories are hiding.

Everything above reflects the Public Preview as it stands today. Given how fast this area is moving, double-check the current docs before you build anything load-bearing on it.

topic Unity Catalog Managed Iceberg Tables: What They Actually Are, and What Tripped Me Up in Community Articles