Unity Catalog federation with Snowflake-managed Iceberg table fails with Universal Format conversion

fhameed
New Contributor

‎07-23-2025 12:07 AM

Hello All,

 
Error:
[DELTA_UNIVERSAL_FORMAT_CONVERSION_FAILED] Failed to convert the table version 3 to the universal format iceberg. Clone validation failed - Size and number of data files in target table should match with source table. srcTableSize: 711784448, targetTableSize: 7296387072 srcTableNumFiles: 421, targetTableNumFiles: 421

Setup:
  • Databricks Unity Catalog federation (not database federation)
  • Snowflake-managed Iceberg table
  • Serverless SQL warehouse
  • Table initially readable, breaks after INSERT in Snowflake

Root Cause Identified:
Metadata version mismatch - Databricks Universal Format stuck converting Iceberg metadata version 3 while Snowflake table has advanced to version 4. Universal Format conversion should NOT be happening on Iceberg federated tables.
 
Failed Workarounds:

  1. Cannot disable Universal Format (config not available in serverless)
  2. REFRESH TABLE triggers same error

Expected Behavior:
Iceberg tables should be read directly without Universal Format conversion attempts.
0 Kudos

mani_22
Databricks Employee
Databricks Employee

‎07-29-2025 01:13 PM

@fhameed  The error occurs if the Iceberg metadata written by Snowflake does not match the number of files in object storage. When attempting to read the table in Databricks, there is a verification process that checks to see if the Iceberg metadata matches exactly the physical files in storage, and if it fails, it throws the error. The issue can occur if the iceberg metadata is corrupted. You can try setting the below configs in classic compute to skip the checksum validation and see if you can query the table.

SET spark.databricks.delta.checksum.mismatch.fatal = false

SET spark.databricks.delta.uniform.ingress.refreshChecksumValidation.enabled = false

0 Kudos

AzBarbarian
New Contributor II
In response to mani_22

‎08-15-2025 08:59 AM

How well is this expected to work?  I have a Snowflake Managed Iceberg table with 150,000,003 rows.  Did a Select count(*) with this via an Interactive Cluster with those settings.  Came back but took 2.5 mins.  I added 1 row to the Snowflake Iceberg table, tried select count(*) again in DBX.  So far I've been waiting over 22 mins, no result yet.

0 Kudos