topic Re: Unity Catalog federation with Snowflake-managed Iceberg table fails with Universal Format conver in Administration & Architecture

Unity Catalog federation with Snowflake-managed Iceberg table fails with Universal Format conversion

fhameed — Wed, 23 Jul 2025 07:07:19 GMT

Hello All,

Error:
[DELTA_UNIVERSAL_FORMAT_CONVERSION_FAILED] Failed to convert the table version 3 to the universal format iceberg. Clone validation failed - Size and number of data files in target table should match with source table. srcTableSize: 711784448, targetTableSize: 7296387072 srcTableNumFiles: 421, targetTableNumFiles: 421

Setup:

Databricks Unity Catalog federation (not database federation)
Snowflake-managed Iceberg table
Serverless SQL warehouse
Table initially readable, breaks after INSERT in Snowflake

Root Cause Identified:
Metadata version mismatch - Databricks Universal Format stuck converting Iceberg metadata version 3 while Snowflake table has advanced to version 4. Universal Format conversion should NOT be happening on Iceberg federated tables.

Failed Workarounds:

Cannot disable Universal Format (config not available in serverless)
REFRESH TABLE triggers same error

Expected Behavior:
Iceberg tables should be read directly without Universal Format conversion attempts.

Re: Unity Catalog federation with Snowflake-managed Iceberg table fails with Universal Format conver

mani_22 — Tue, 29 Jul 2025 20:13:21 GMT

@fhameed The error occurs if the Iceberg metadata written by Snowflake does not match the number of files in object storage. When attempting to read the table in Databricks, there is a verification process that checks to see if the Iceberg metadata matches exactly the physical files in storage, and if it fails, it throws the error. The issue can occur if the iceberg metadata is corrupted. You can try setting the below configs in classic compute to skip the checksum validation and see if you can query the table.

SET spark.databricks.delta.checksum.mismatch.fatal = false

SET spark.databricks.delta.uniform.ingress.refreshChecksumValidation.enabled = false

Re: Unity Catalog federation with Snowflake-managed Iceberg table fails with Universal Format conver

AzBarbarian — Fri, 15 Aug 2025 15:59:36 GMT

How well is this expected to work? I have a Snowflake Managed Iceberg table with 150,000,003 rows. Did a Select count(*) with this via an Interactive Cluster with those settings. Came back but took 2.5 mins. I added 1 row to the Snowflake Iceberg table, tried select count(*) again in DBX. So far I've been waiting over 22 mins, no result yet.