topic Re: Setting up a snowflake catalog via spark config next to unity catalog in Warehousing & Analytics

Setting up a snowflake catalog via spark config next to unity catalog

Laurens — Wed, 27 Dec 2023 13:10:15 GMT

Im trying to set up a connection to Iceberg on S3 via Snowflake as described https://medium.com/snowflake/how-to-integrate-databricks-with-snowflake-managed-iceberg-tables-7a8895c2c724 and https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog

On a cluster with Unity Catalog disabled this works fine and I can read from the tables without connecting to a snowflake warehouse. When Unity Catalog is enabled however, the `snowflake_catalog` is nowhere to be found.

Is there a way to setup a catalog via Spark config like described above while using Unity Catalog?

Thanks in advance!

Re: Setting up a snowflake catalog via spark config next to unity catalog

Laurens — Wed, 20 Mar 2024 13:15:44 GMT

Hi Kaniz,

Thanks for your elaborate reply! The iceberg data that we have to interact with uses Snowflake as catalog, not glue. Can we also use a catalog integration with Snowflake?

Thanks!

Re: Setting up a snowflake catalog via spark config next to unity catalog

Laurens — Thu, 04 Apr 2024 07:13:27 GMT

Hi @Retired_mod ,

We've been working on setting up Glue as catalog, which is working fine so far. However, Glue takes place of the hive_metastore, which appears to be a legacy way of setting this up. Is the way proposed here the recommended way to set it up, or is there a more "unity catalog" way of setting it up? https://docs.databricks.com/en/archive/external-metastores/aws-glue-metastore.html

Thanks in advance,

Laurens