Setting up a snowflake catalog via spark config next to unity catalog

[Laurens]

Im trying to set up a connection to Iceberg on S3 via Snowflake as described https://medium.com/snowflake/how-to-integrate-databricks-with-snowflake-managed-iceberg-tables-7a889... and https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog

On a cluster with Unity Catalog disabled this works fine and I can read from the tables without connecting to a snowflake warehouse. When Unity Catalog is enabled however, the `snowflake_catalog` is nowhere to be found.

Is there a way to setup a catalog via Spark config like described above while using Unity Catalog?

Thanks in advance!

[Kaniz]

Hi @Laurens, Integrating Iceberg tables on S3 with Snowflake can be done effectively, even when using the Unity Catalog.

Let’s break down the steps to achieve this:

1. Configure an External Volume to Amazon S3:

   • Set up an external volume that points to the location of your Iceberg tables on Amazon S3. This volume will grant Snowflake read-only access to the Iceberg data.

2. Catalog Integration with AWS Glue:

   • Create a catalog integration with AWS Glue. Ensure that the integration has the necessary permissions to access the database containing your Iceberg tables. This step allows Snowflake to seamlessly interact with the Iceberg metadata.

3. Create Iceberg Tables:

   • Now you can create Iceberg tables within Snowflake, using AWS Glue as the catalog. These tables will be linked to the Iceberg datasets stored in S3.

By following these steps, you’ll be able to query your Iceberg datasets from Snowflake, even with Unity Catalog enabled. The integration ensures that you can work with Iceberg tables efficiently while benefiting from the power of Snowflake’s data platform. 🌟

For more detailed information, you can refer to the resources you mentioned:

• Unifying Iceberg Tables on Snowflake¹.
• Medium Article: Migrate Existing Amazon S3 Data Lake Tables to Snowflake Unmanaged Iceberg Tables Us...².

Feel free to explore these links for deeper insights into the process. If you have any further questions, don’t hesitate to ask! 😊

[Laurens]

Hi Kaniz,

Thanks for your elaborate reply! The iceberg data that we have to interact with uses Snowflake as catalog, not glue. Can we also use a catalog integration with Snowflake?

Thanks!

[Laurens]

Hi @Kaniz ,

We've been working on setting up Glue as catalog, which is working fine so far. However, Glue takes place of the hive_metastore, which appears to be a legacy way of setting this up. Is the way proposed here the recommended way to set it up, or is there a more "unity catalog" way of setting it up? https://docs.databricks.com/en/archive/external-metastores/aws-glue-metastore.html

Thanks in advance,

Laurens