https://community.databricks.com/t5/technical-blog/how-to-set-up-databricks-snowflake-interoperability-with-unity/ba-p/128406 <P><SPAN>In this post, we’ll explore the two main approaches to integrating UC with other Iceberg environments — </SPAN><STRONG>Foreign Iceberg tables</STRONG><SPAN> and </SPAN><STRONG>Managed Iceberg tables </STRONG><SPAN>— and walk through when to choose each, how to configure them, and the operational considerations to keep in mind.</SPAN></P> <P><SPAN>Seamless interoperability between </SPAN><STRONG>Databricks Unity Catalog (UC)</STRONG><SPAN> and external Apache Iceberg ecosystems, such as </SPAN><STRONG>Snowflake Horizon</STRONG><SPAN>, enables organizations to use the right engine for the right workload without being locked into a single technology stack. By standardizing on Iceberg and establishing the right catalog strategy, teams can achieve </SPAN><STRONG>governed, engine-agnostic data sharing</STRONG><SPAN> while avoiding costly and unnecessary data duplication.</SPAN><SPAN>&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Screenshot 2025-08-14 at 5.43.13 PM.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/19103iB4D9E1BCE51C6207/image-size/large?v=v2&amp;px=999" role="button" title="Screenshot 2025-08-14 at 5.43.13 PM.png" alt="Screenshot 2025-08-14 at 5.43.13 PM.png" /></span></P> <P>&nbsp;</P> <H2><STRONG>Understanding Apache Iceberg and Catalogs</STRONG></H2> <P><SPAN>Apache Iceberg is an </SPAN><STRONG>open table format</STRONG><SPAN> for large-scale data lakes that supports ACID transactions, schema evolution, and branching. However, Iceberg requires a </SPAN><STRONG>catalog</STRONG><SPAN> to track table metadata, coordinate transactions, and enforce governance.</SPAN></P> <P><SPAN>A catalog is responsible for:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Discovery</STRONG><SPAN> – knowing where tables live and what metadata they reference.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Concurrency control</STRONG><SPAN> – managing atomic snapshot updates across engines.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Governance</STRONG><SPAN> – controlling permissions, lineage, and auditing.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Optimization hooks</STRONG><SPAN> – triggering compaction, clustering, and snapshot expiration.</SPAN><SPAN><BR /></SPAN></LI> </UL> <P><SPAN>In a multi-engine architecture, </SPAN><STRONG>the choice of catalog becomes just as important as the table format itself</STRONG><SPAN>. The long-term vision for an open lakehouse is to </SPAN><STRONG>decouple compute from storage</STRONG><SPAN> so you can choose the best engine for the workload—while letting the catalog define governance, lifecycle, and optimization.</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Managed Tables (UC-owned):</STRONG><SPAN> Unity Catalog owns the table’s lifecycle — handling reads, writes, and automated maintenance such as Predictive Optimization.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Foreign Tables (Externally-owned):</STRONG><SPAN> An external catalog, such as Snowflake Horizon, manages the lifecycle. Unity Catalog provides secure, governed read access without taking ownership of the table.</SPAN></LI> </UL> <H3><STRONG>Choosing the Right Approach — Quick Reference</STRONG></H3> <TABLE> <THEAD> <TR> <TH> <P><STRONG>Requirement</STRONG></P> </TH> <TH> <P><STRONG>Pick</STRONG></P> </TH> </TR> </THEAD> <TBODY> <TR> <TD> <P><SPAN>Only </SPAN><STRONG>read</STRONG><SPAN> in Databricks; lifecycle owned elsewhere</SPAN></P> </TD> <TD> <P><STRONG>Foreign Iceberg (Catalog Federation)</STRONG></P> </TD> </TR> <TR> <TD> <P><SPAN>Full </SPAN><STRONG>read/write</STRONG><SPAN> from Databricks </SPAN><STRONG>and</STRONG><SPAN> other engines, with UC governance &amp; automation</SPAN></P> </TD> <TD> <P><STRONG><span class="lia-unicode-emoji" title=":white_heavy_check_mark:">✅</span> Managed Iceberg + UC REST Catalog</STRONG></P> </TD> </TR> <TR> <TD> <P><SPAN>Need </SPAN><STRONG>automated maintenance</STRONG><SPAN> (optimize, expire snapshots)</SPAN></P> </TD> <TD> <P><STRONG><span class="lia-unicode-emoji" title=":white_heavy_check_mark:">✅</span> Managed Iceberg + Predictive Optimization</STRONG></P> </TD> </TR> <TR> <TD> <P><SPAN>Need to </SPAN><STRONG>evolve clustering strategy without rewrite</STRONG></P> </TD> <TD> <P><STRONG><span class="lia-unicode-emoji" title=":white_heavy_check_mark:">✅</span> Managed Iceberg + Liquid Clustering</STRONG></P> </TD> </TR> </TBODY> </TABLE> <P>&nbsp;</P> <H2><STRONG>Two Approaches to Iceberg Interoperability in UC</STRONG></H2> <H3><STRONG>1. Foreign Iceberg Tables (read-only from Databricks)</STRONG></H3> <P><SPAN>Foreign Iceberg tables are </SPAN><STRONG>owned and maintained by an external catalog </STRONG><SPAN>— for example, Snowflake Horizon or AWS Glue — but made visible in Unity Catalog. This approach is ideal when Databricks needs </SPAN><STRONG>read-only access</STRONG><SPAN> to tables whose lifecycle is managed elsewhere.</SPAN></P> <P><SPAN>The most robust way to access foreign Iceberg is </SPAN><STRONG>Catalog Federation</STRONG><SPAN>, where UC connects directly to the external catalog and mirrors table metadata. Each query checks for the latest snapshot from the foreign catalog before execution, ensuring freshness without losing UC governance, lineage tracking, and audit visibility.</SPAN></P> <H4><SPAN>Configuring Foreign Iceberg in UC</SPAN></H4> <P><STRONG>Step 1.</STRONG><A href="https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/external-locations#-option-2-create-an-external-location-manually-using-catalog-explorer" target="_blank" rel="noopener"> <STRONG>Create</STRONG></A><STRONG> External Location(s)</STRONG><SPAN> (authorized paths to the Iceberg data)</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Lets UC read the cloud storage referenced by the foreign catalog.</SPAN></LI> </UL> <P><STRONG>Step 2.</STRONG><A href="https://docs.databricks.com/aws/en/query-federation/snowflake#create-a-connection" target="_blank" rel="noopener"> <STRONG>Create</STRONG></A><STRONG> a Connection to the Foreign Catalog</STRONG></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>E.g., Snowflake Horizon: connection must have </SPAN><STRONG>USAGE</STRONG><SPAN> on database, schema, external volume, and the Iceberg table(s) to resolve current metadata locations.</SPAN></LI> </UL> <P><STRONG>Step 3.</STRONG><A href="https://docs.databricks.com/aws/en/query-federation/hms-federation-external#-create-the-foreign-catalog" target="_blank" rel="noopener"> <STRONG>Create</STRONG></A><STRONG> a Federated Catalog in UC</STRONG></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Point it at the connection + </SPAN><STRONG>authorize</STRONG><SPAN> it to the external locations from Step 1.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Every query checks for metadata freshness before proceeding.</SPAN></LI> </UL> <LI-CODE lang="php">SELECT * FROM federated_catalog.federated_schema.iceberg_table</LI-CODE> <P><SPAN>&nbsp;</SPAN></P> <H3><STRONG>2. Managed Iceberg Tables (full interoperability, read/write)</STRONG></H3> <P><SPAN>Managed Iceberg tables are </SPAN><STRONG>created and governed by Unity Catalog</STRONG><SPAN>. UC is the system of record, and all engines — including Snowflake, Trino, Flink, or Spark — access the tables through the </SPAN><STRONG>UC Iceberg REST Catalog</STRONG><SPAN>. This setup supports </SPAN><STRONG>full read/write interoperability</STRONG><SPAN> while preserving UC’s centralized governance.</SPAN></P> <P><SPAN>Managed Iceberg offers:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Single source of truth</STRONG><SPAN> for data and metadata.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Predictive Optimization</STRONG><SPAN> to automatically handle compaction, snapshot expiration, and other maintenance tasks.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Liquid Clustering</STRONG><SPAN> for evolving data layout strategies without rewriting historical data.</SPAN></LI> </UL> <H4><SPAN>Configuring Managed Iceberg in UC</SPAN></H4> <P><STRONG>Step 1. Create the table</STRONG></P> <LI-CODE lang="php">CREATE OR REPLACE TABLE main.schema.iceberg_table (c1 INT) USING iceberg;</LI-CODE> <P><SPAN>(or)</SPAN></P> <LI-CODE lang="python">df.write.format("iceberg").saveAsTable("main.schema.iceberg_table")</LI-CODE> <P><SPAN>If you try to specify a </SPAN><STRONG>LOCATION</STRONG><SPAN> on a managed Iceberg table, Databricks will </SPAN><STRONG>error </STRONG><SPAN>— UC manages the storage.</SPAN></P> <P><STRONG>Step 2 (Optional but recommended) Enable Liquid Clustering</STRONG></P> <LI-CODE lang="php">ALTER TABLE main.schema.iceberg_table CLUSTER BY (c1);</LI-CODE> <P><STRONG>Note:</STRONG><SPAN> Currently, to make </SPAN><SPAN>CLUSTER BY</SPAN><SPAN> work on Iceberg tables in UC, you must first set the following table properties:</SPAN></P> <LI-CODE lang="ruby">ALTER TABLE main.schema.iceberg_table SET TBLPROPERTIES ( 'delta.enableDeletionVectors' = false, 'delta.enableRowTracking' = false );</LI-CODE> <P><SPAN>This extra step will </SPAN><STRONG>not</STRONG><SPAN> be required in the near future — </SPAN><STRONG>with Iceberg v3</STRONG><SPAN> you’ll be able to simply run </SPAN><SPAN>CLUSTER BY</SPAN><SPAN> without the table property configuration.</SPAN></P> <P><STRONG>Step 3. Use standard DML</STRONG></P> <LI-CODE lang="php">INSERT INTO main.schema.iceberg_table VALUES (11);</LI-CODE> <P><STRONG>Step 4. Enable external data access (metastore level)</STRONG></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>UC vends short-lived credentials so external engines can access the underlying storage.</SPAN></LI> </UL> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Screenshot 2025-08-14 at 12.00.53 PM.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/19101i5FD1F5AD170883E1/image-size/medium?v=v2&amp;px=400" role="button" title="Screenshot 2025-08-14 at 12.00.53 PM.png" alt="Screenshot 2025-08-14 at 12.00.53 PM.png" /></span><BR /></SPAN></P> <P><STRONG>Step 5. Grant </STRONG><STRONG>EXTERNAL USE SCHEMA</STRONG><SPAN> (it’s </SPAN><STRONG>not</STRONG><SPAN> in </SPAN><SPAN>ALL PRIVILEGES</SPAN><SPAN>)</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Grant at catalog or schema level to the service principal / user.</SPAN></LI> </UL> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Screenshot 2025-08-14 at 11.59.06 AM.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/19102i4F2AE6530D0E437D/image-size/medium?v=v2&amp;px=400" role="button" title="Screenshot 2025-08-14 at 11.59.06 AM.png" alt="Screenshot 2025-08-14 at 11.59.06 AM.png" /></span></SPAN></P> <P><STRONG>Step 6. Configure the Iceberg REST Catalog client</STRONG></P> <P><STRONG>Auth Option A — Service Principal</STRONG></P> <LI-CODE lang="php">CREATE OR REPLACE CATALOG INTEGRATION unity_catalog CATALOG_SOURCE = ICEBERG_REST TABLE_FORMAT = ICEBERG CATALOG_NAMESPACE = '&lt;uc-schema name&gt;' REST_CONFIG = ( CATALOG_URI = 'https://&lt;workspace-uri&gt;/api/2.1/unity-catalog/iceberg-rest' WAREHOUSE = '&lt;uc-catalog name&gt;' ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS ) REST_AUTHENTICATION = ( TYPE = OAUTH OAUTH_TOKEN_URI = 'https://&lt;workspace-uri&gt;/oidc/v1/token' OAUTH_CLIENT_ID = '&lt;client-id&gt;' OAUTH_CLIENT_SECRET = '&lt;client-secret&gt;' OAUTH_ALLOWED_SCOPES = ('all-apis') ) ENABLED = TRUE;</LI-CODE> <P><STRONG>Auth Option B — Personal Access Token</STRONG></P> <LI-CODE lang="php">CREATE OR REPLACE CATALOG INTEGRATION unity_catalog CATALOG_SOURCE = ICEBERG_REST TABLE_FORMAT = ICEBERG CATALOG_NAMESPACE = '&lt;uc-schema-name&gt;' REST_CONFIG = ( CATALOG_URI = 'https://&lt;workspace-uri&gt;/api/2.1/unity-catalog/iceberg-rest' WAREHOUSE = '&lt;uc-catalog-name&gt;' ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS ) REST_AUTHENTICATION = ( TYPE = BEARER BEARER_TOKEN = '&lt;personal access token&gt;' ) ENABLED = TRUE;</LI-CODE> <P><STRONG><span class="lia-unicode-emoji" title=":information:">ℹ️</span> Additional note for Snowflake writes</STRONG></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Vended credentials</STRONG><SPAN> currently </SPAN><STRONG>work for reads only</STRONG><SPAN> from Snowflake into UC-managed Iceberg.</SPAN><SPAN><BR /></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>To </SPAN><STRONG>write</STRONG><SPAN> to a UC-managed Iceberg table from Snowflake:</SPAN><SPAN><BR /></SPAN></LI> <OL> <LI style="font-weight: 400;" aria-level="2"><SPAN>Create an </SPAN><STRONG>external location</STRONG><SPAN> on the Snowflake side pointing to the UC table’s underlying storage.</SPAN></LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>Create the </SPAN><STRONG>catalog integration</STRONG><SPAN> in Snowflake the same way as for reads, </SPAN><STRONG>but omit</STRONG> <SPAN>ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS</SPAN><SPAN>.</SPAN></LI> <LI style="font-weight: 400;" aria-level="2"><SPAN>When creating the Iceberg table in Snowflake, </SPAN><STRONG>explicitly specify the external location</STRONG><SPAN>.</SPAN><SPAN><BR /><BR /></SPAN></LI> </OL> </UL> <H2><STRONG>Common Pitfalls to Avoid</STRONG></H2> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Missing authorized paths</STRONG><SPAN> in federated catalogs will cause queries to fall back to query federation, potentially incurring double compute costs.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Forgetting </STRONG><STRONG>EXTERNAL USE SCHEMA</STRONG><SPAN> when granting external engines access will block reads/writes. It’s </SPAN><STRONG>not</STRONG><SPAN> implied by </SPAN><SPAN>ALL PRIVILEGES</SPAN><SPAN>.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Not setting a root storage location</STRONG><SPAN> for a federated catalog in UC; this is required to persist UC’s metadata about the foreign catalog. This is </SPAN><STRONG>separate from authorized paths</STRONG><SPAN>.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Version requirements</STRONG><SPAN>: DBR 16.4 LTS+ for reads and writes.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Not enabling Predictive Optimization</STRONG><SPAN> can lead to performance degradation over time.</SPAN><SPAN><BR /></SPAN><SPAN>&nbsp;</SPAN></LI> </UL> <H2><STRONG>Conclusion</STRONG></H2> <P><SPAN>Apache Iceberg’s open, engine-agnostic design enables true cross-platform interoperability—but </SPAN><STRONG>only if paired with the right catalog</STRONG><SPAN> strategy. By deciding early whether a dataset should be treated as </SPAN><STRONG>Foreign Iceberg</STRONG><SPAN> or </SPAN><STRONG>Managed Iceberg</STRONG><SPAN>, and by following best practices for federation, REST Catalog integration, and governance, you can build a lakehouse architecture that supports multiple engines without sacrificing performance, security, or maintainability.</SPAN></P> Wed, 12 Nov 2025 16:54:15 GMT MohanaBasak 2025-11-12T16:54:15Z https://community.databricks.com/t5/technical-blog/how-to-set-up-databricks-snowflake-interoperability-with-unity/ba-p/128406 <P>Unlock seamless interoperability between Databricks Unity Catalog and external Iceberg ecosystems like Snowflake Horizon — without data copies. Learn when to use Managed vs. Foreign Iceberg, how to enable read/write access across platforms, and the pitfalls to avoid.</P> Wed, 12 Nov 2025 16:54:15 GMT https://community.databricks.com/t5/technical-blog/how-to-set-up-databricks-snowflake-interoperability-with-unity/ba-p/128406 MohanaBasak 2025-11-12T16:54:15Z