cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Now that Unity Catalog manages Iceberg natively, is anyone actually switching?

Islam_hoti
New Contributor II

Hi everyone,

Unity Catalog managed Iceberg tables went GA earlier this year, and they arrived with the features that used to be the reason to stay on Delta: liquid clustering, predictive optimization, materialized views, streaming tables. On paper the gap is basically closed.

What I find interesting is that this quietly changes the shape of the argument. For years the choice was framed as Delta if you live on Databricks, Iceberg if you need everything else to read your tables. That tradeoff is mostly gone now, which means the decision stops being technical and starts being political. Who else needs to read this data, and how much do you trust that to stay true in three years?

So I am curious what people are actually doing rather than what the blog posts say.

Have you created any managed Iceberg tables in production yet, or is it still a proof of concept?

If you did move, what pushed you? A specific engine that had to read the data, a procurement or vendor lock-in conversation, or something more practical?

If you stayed on Delta, was that a real evaluation or just inertia? No judgement, inertia is a legitimate engineering strategy.

For anyone running both, has the operational overhead of two formats been worth it, or did you regret the split?

And the one I am most curious about: has anyone hit a case where managed Iceberg behaved differently from Delta in a way that actually cost you time?

My own read is that most teams will stay on Delta simply because nothing is forcing them off it, and that Iceberg adoption will be driven almost entirely by organizations that have a second engine in the picture. Happy to be told I am wrong.

What is your team doing?

1 ACCEPTED SOLUTION

Accepted Solutions

Khasim_1
New Contributor II

Hi @Islam_hoti,

The Reality on the Ground: We have experimented with Iceberg, but for our primary production workloads, we remain 100% Delta Lake. Here is our internal "Messy Truth" on why:

  1. The "Inertia" Factor: You’re right, inertia is a legitimate engineering strategy. When Delta Lake is already fully integrated with Unity Catalog, DLT, and Serverless compute, it’s not just a "format"; it’s the default data-plane of our entire platform. The operational risk of managing two formats is simply higher than the benefit of switching.
  2. The "Second Engine" Driver: We only see teams move to Iceberg when they have a critical, non-negotiable requirement for an engine outside of the Databricks ecosystem that lacks robust Delta support. If you are a "Databricks-first" shop, the "native" feel of Delta Lake—especially with features like Liquid Clustering and Predictive Optimization—is still just a bit more "frictionless" than Iceberg.
  3. The Hidden Cost of "Both": We once looked at running both, but the operational overhead of maintaining two different sets of DLT pipelines and governance workflows was a non-starter. It doubles your testing matrix for zero incremental gain in performance.

 

if you’re looking for the official best practices on workspace/catalog isolation, I’d highly recommend checking out the https://docs.databricks.com/en/tables/iceberg.html, which breaks down the trade-offs nicely."

Data Architect | 13 Years Domain Expertise | Databricks SA Champion Cohort

View solution in original post

1 REPLY 1

Khasim_1
New Contributor II

Hi @Islam_hoti,

The Reality on the Ground: We have experimented with Iceberg, but for our primary production workloads, we remain 100% Delta Lake. Here is our internal "Messy Truth" on why:

  1. The "Inertia" Factor: You’re right, inertia is a legitimate engineering strategy. When Delta Lake is already fully integrated with Unity Catalog, DLT, and Serverless compute, it’s not just a "format"; it’s the default data-plane of our entire platform. The operational risk of managing two formats is simply higher than the benefit of switching.
  2. The "Second Engine" Driver: We only see teams move to Iceberg when they have a critical, non-negotiable requirement for an engine outside of the Databricks ecosystem that lacks robust Delta support. If you are a "Databricks-first" shop, the "native" feel of Delta Lake—especially with features like Liquid Clustering and Predictive Optimization—is still just a bit more "frictionless" than Iceberg.
  3. The Hidden Cost of "Both": We once looked at running both, but the operational overhead of maintaining two different sets of DLT pipelines and governance workflows was a non-starter. It doubles your testing matrix for zero incremental gain in performance.

 

if you’re looking for the official best practices on workspace/catalog isolation, I’d highly recommend checking out the https://docs.databricks.com/en/tables/iceberg.html, which breaks down the trade-offs nicely."

Data Architect | 13 Years Domain Expertise | Databricks SA Champion Cohort