article Integrating Apache Spark™ with Databricks Unity Catalog Assets via Open APIs in Technical Blog

Integrating Apache Spark™ with Databricks Unity Catalog Assets via Open APIs

dkushari — Fri, 05 Jun 2026 09:19:34 GMT

Re: Integrating Apache Spark™ with Databricks Unity Catalog Assets via Open APIs

Klugscheißer — Thu, 12 Dec 2024 21:11:29 GMT

This looks exactly like what I've been hoping for! However, the tutorial as shown here doesn't seem to work for me.

The repo https://s01.oss.sonatype.org/content/repositories/iounitycatalog-1016/ does not exist and 404s. Luckily, unitycatalog-spark is published elsewhere on sonatype
There seems to be a version mismatch in the dependent hadoop libraries.
- Spark 3.5.{0-3} are all compiled against hadoop 3.3.4.
- unitycatalog-spark versions 0.2.{0,1} depend on spark 3.5.3.
- unitycatalog-spark also depend on hadoop-client-runtime 3.4.0
- hadoop libraries 3.4.0 have an updated FSBuilder that has an `optLong` method that is not present in 3.3.4. This results in `java.lang.NoSuchMethodError` if this tutorial is attempted with spark 3.3.4 and `java.lang.NumberFormatException` if upgraded to 3.4.0.

Is this tutorial possible or was unitycatalog-spark released with incompatible dependencies? Thanks!

Re: Integrating Apache Spark™ with Databricks Unity Catalog Assets via Open APIs

Klugscheißer — Fri, 13 Dec 2024 21:39:34 GMT

I was able to resolve the above dependency issues by explicitly depending on hadoop-aws 3.4.1 in addition to `spark-hadoop-cloud` and was able to read and write to tables in UC from OSS spark!

However, I'm now having difficulty with the Uniform part. If I only add a dependency on `iceberg-spark-runtime-3.5`, then writing to a Uniform iceberg table fails with

ERROR org.apache.spark.sql.delta.DeltaLog - Failed to find Iceberg converter class
java.lang.ClassNotFoundException: org.apache.spark.sql.delta.icebergShaded.IcebergConverter

If I add `delta-iceberg` then the IcebergConverter class is found, reading and writing to the table works, however the iceberg conversion on write fails with

ERROR org.apache.spark.sql.delta.icebergShaded.IcebergConverter - Error when converting to Iceberg metadata
org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: [SCHEMA_NOT_FOUND] The schema `test_schema` cannot be found. Verify the spelling and correctness of the schema and catalog.
[info] If you did not qualify the name with a catalog, verify the current_schema() output, or qualify the name with the correct catalog.

Possibly related, while `SHOW SCHEMAS` does list the `test_schema` I've been using and `SELECT current_schema()` does indeed return `test_schema`, `spark.sessionState.catalog.listDatabases()` does not! Reading the code, it seems like this is what's referenced during the iceberg conversion. It's not clear to me why the schema / db is not consistent across various spark calls.

Re: Integrating Apache Spark™ with Databricks Unity Catalog Assets via Open APIs

sparkplug — Tue, 09 Dec 2025 12:58:50 GMT

That's a great article, thanks for sharing! It perfectly aligns with our requirement for a more open approach.

However, I have a follow-up question regarding Access Tokens from Entra for Databricks authentication to Unity Catalog when running Spark on Kubernetes (K8s).

Specifically, I'm encountering an issue with long-running tasks (over 1 hour):

How can I configure the Spark session to automatically refresh these tokens?

Re: Integrating Apache Spark™ with Databricks Unity Catalog Assets via Open APIs

dbx_8451 — Tue, 16 Dec 2025 18:13:19 GMT

Could you clarify how you are leveraging credential vending? I noticed that the blog post appears to use the Databricks PAT directly, and I didn’t see credential vending mentioned. Could you please explain?

=====
With its open APIs and introduction of credential vending, Databricks Unity Catalog data can be read by external engines and interfaces such as Iceberg REST APIs, DuckDB, Apache Spark™, Trino.

/spark-sql --name "s3-uc-test" \
	--master "local[*]" \
	--packages "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.6.1,io.delta:delta-spark_2.12:3.2.1,io.unitycatalog:unitycatalog-spark_2.12:0.2.0,org.apache.hadoop:hadoop-common:3.3.4,org.apache.hadoop:hadoop-aws:3.3.4" \
	--conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
	--conf "spark.sql.catalog.spark_catalog=io.unitycatalog.spark.UCSingleCatalog" \
	--conf "spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" \
	--conf "spark.sql.catalog.<<Your Default UC Catalog>>=io.unitycatalog.spark.UCSingleCatalog" \
	--conf "spark.sql.catalog.<<Your Default UC Catalog>>.uri=https://<<Your Databricks Workspace URL>>/api/2.1/unity-catalog" \
	--conf "spark.sql.catalog.<<Your Default UC Catalog>>.token=<<Your Databricks Workspace PAT>>" \
	--conf "spark.sql.defaultCatalog=<<Your Default UC Catalog>>"