UC Volume access for spark and other config files

dwfchu1 — Wed, 10 Jan 2024 07:34:28 GMT

Hi All,

Wondering if anyone else getting this problem:

We trying to host krb5.conf and jaas.conf for our compute to be able to connect to Kerberised JDBC sources, we attempting to store these files in Catalog volumes, but at run time when initiating the connection with spark.read.format("jdbc"), we get the following error:

"Cannot access the UC Volume path from this location. Path was...."

We using 14.2 (includes Apache Spark 3.5.0, Scala 2.12) runtime in Single User mode, which should be able to connect to UC volumes.

Example config we are using:

Djava.security.auth.login.config=dbfs:/Volumes/.../.../jaas.conf

We also checked that the single user also has ALL PRIVILEGES permissions on the catalog and subsequent schema/volumes

Also adding:

spark.databricks.unityCatalog.volumes.enabled true

Seems to make no difference 😞

We have also tested that the connection works successfully with just having this in DBFS natively, but ideally we'd like to have better organisation and access-controlled of these files (keytab specifically) per catalog. Thanks!

Re: UC Volume access for spark and other config files

mbendana — Fri, 12 Jan 2024 20:16:04 GMT

Haven't been able to access volume path when using jdbc format.

topic UC Volume access for spark and other config files in Data Engineering

UC Volume access for spark and other config files

Re: UC Volume access for spark and other config files