Is there a way to natively mount external Iceberg REST Catalogs (e.g., BigLake) in Unity Catalog?

[ismaelhenzel]

Hi everyone,

I have been reviewing the documentation on integrating external Iceberg tables with Databricks. Currently, the only method I have found to read from an Iceberg REST catalog (specifically GCP BigLake in my case) is by explicitly passing the catalog configurations directly to the SparkSession.

Here is the approach I am currently using, which works successfully on standard clusters:

import pyspark
from pyspark.context import SparkContext
from pyspark.sql import SparkSession

# Define specific variables
spark_catalog = "spark_catalog"
gcp_project_id = "mul-dev-databricks"
warehouse_path = f"bq://projects/{gcp_project_id}/locations/us"
gcp_scopes = "https://www.googleapis.com/auth/cloud-platform"

# Build the SparkSession with REST catalog configs
spark = SparkSession.builder.appName("BigLake_Iceberg_App") \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config('spark.sql.catalog.spark_catalog', 'org.apache.iceberg.spark.SparkCatalog') \
    .config('spark.sql.catalog.spark_catalog.type', 'rest') \
    .config('spark.sql.catalog.spark_catalog.uri', 'https://biglake.googleapis.com/iceberg/v1/restcatalog') \
    .config('spark.sql.catalog.spark_catalog.warehouse', warehouse_path) \
    .config('spark.sql.catalog.spark_catalog.header.x-goog-user-project', gcp_project_id) \
    .config("spark.sql.catalog.spark_catalog.gcp.auth.scopes", gcp_scopes) \
    .config('spark.sql.catalog.spark_catalog.header.X-Iceberg-Access-Delegation', 'vended-credentials') \
    .config('spark.sql.catalog.spark_catalog.rest.auth.type', 'org.apache.iceberg.gcp.auth.GoogleAuthManager') \
    .config('spark.sql.catalog.spark_catalog.io-impl', 'org.apache.iceberg.gcp.gcs.GCSFileIO') \
    .config('spark.sql.catalog.spark_catalog.rest-metrics-reporting-enabled', 'false') \
    .config('spark.sql.defaultCatalog', 'spark_catalog') \
    .getOrCreate()

While I can successfully access the data this way, it is not an ideal long-term solution. It would be much more powerful if Unity Catalog could mount these REST catalogs directly as external catalogs natively (similar to Unity Catalog Federation).

Relying on Spark configurations introduces a few key challenges:

1. Serverless Limitations: Serverless compute has strict limitations regarding custom Spark configurations. Setting these at the cluster level completely blocks us from utilizing Serverless SQL/Compute for this data.

2. Governance: Managing connections via code or cluster configs bypasses the centralized governance and access control benefits of Unity Catalog.

3. User Experience: It requires every user or job to duplicate these boilerplate configurations.

My questions to the community:

• Is there currently a supported way to configure an Iceberg REST Catalog connection directly inside Unity Catalog as a Foreign/External Catalog?

• If not, is there an alternative workaround that plays nicely with Databricks Serverless?

• Is native Unity Catalog federation for Iceberg REST catalogs on the Databricks roadmap?

Thanks in advance for any insights!

[amirabedhiafi · Accepted solution]

Hello @ismaelhenzel 

AFAIK, there is no documented native UC foreign catalog integration for a generic Iceberg REST catalog such as BigLake REST Catalog today.

DBKS does support Iceberg in UC including UC managed and foreign Iceberg tables but the documented foreign Iceberg support is through Lakehouse Federation with supported external catalogs with examples such as AWS Glue, Hive metastore or Snowflake Horizon Catalog.

BigLake REST Catalog is not listed as a native UC catalog federation target. https://docs.databricks.com/gcp/en/iceberg/

For GCP, the closest option is Lakehouse Federation to BigQuery where you create a UC connection and a foreign catalog of type bigquery and this works with SQL warehouses also serverless SQL warehouses and gives you UC level discovery and access control. 

CREATE CONNECTION my_bq_connection TYPE bigquery
OPTIONS (
  GoogleServiceAccountKeyJson secret('scope', 'bq-service-account-json')
);

CREATE FOREIGN CATALOG my_bq_catalog
USING CONNECTION my_bq_connection
OPTIONS (
  dataProjectId 'my-gcp-project'
);

However it is not a native Iceberg REST catalog mount. If your BigLake or Iceberg data is exposed through BigQuery as external tables or views this may be a practical workaround but don't forget that views and external tables are materialized by the BigQuery connector, so behavior (also with perf or cost) may differ from reading the Iceberg REST catalog directly. 

For the pure Iceberg REST approach you showed, I think you are currently limited to classic compute where you can control the Spark catalog configuration. Serverless has restrictions around external data access, Maven data sources, compute scoped libraries and most Spark compute configs so it is not a good fit for this kind of custom Spark catalog setup.

---