https://community.databricks.com/t5/administration-architecture/databricks-apache-iceberg-advantageous-or-wasted-effort-due-to/m-p/81366#M1445 <P>Trying to design a Lakehouse. Spark is at the base layer. Now wondering if adding Apache Iceberg sitting below Spark will be of help, or, not ? Preferring Iceberg for its auto indexing, ACID query facilities over big hetergenous datasets. Wonder if its a wise choice??&nbsp;&nbsp;</P> Wed, 31 Jul 2024 18:40:35 GMT ag2all 2024-07-31T18:40:35Z https://community.databricks.com/t5/administration-architecture/databricks-apache-iceberg-advantageous-or-wasted-effort-due-to/m-p/81366#M1445 <P>Trying to design a Lakehouse. Spark is at the base layer. Now wondering if adding Apache Iceberg sitting below Spark will be of help, or, not ? Preferring Iceberg for its auto indexing, ACID query facilities over big hetergenous datasets. Wonder if its a wise choice??&nbsp;&nbsp;</P> Wed, 31 Jul 2024 18:40:35 GMT https://community.databricks.com/t5/administration-architecture/databricks-apache-iceberg-advantageous-or-wasted-effort-due-to/m-p/81366#M1445 ag2all 2024-07-31T18:40:35Z https://community.databricks.com/t5/administration-architecture/databricks-apache-iceberg-advantageous-or-wasted-effort-due-to/m-p/82015#M1497 <P>Hello, if you're planning on building your own open source stack of spark+iceberg, it can be a good choice.&nbsp;</P> <P>If you're on Databricks, however, you're going to miss out a *lot* on delta features that are baked into the platform. Specifically compute + storage performance based optimisations and UC integrations. Delta has ACID compliance, works beautifully with large datasets and you have many performance choices with liquid clustering or legacy z ordering.&nbsp;</P> <P>If you're integrating with other systems that are only iceberg compatible, check out uniform to write out additional metadata so other systems can read from it:&nbsp;<A href="https://docs.databricks.com/en/delta/uniform.html" target="_blank">https://docs.databricks.com/en/delta/uniform.html</A></P> Tue, 06 Aug 2024 10:10:06 GMT https://community.databricks.com/t5/administration-architecture/databricks-apache-iceberg-advantageous-or-wasted-effort-due-to/m-p/82015#M1497 holly 2024-08-06T10:10:06Z