topic Re: Maven libraries in VNet injected, UC enabled workspace on Standard Access Mode Cluster in Data Engineering

Maven libraries in VNet injected, UC enabled workspace on Standard Access Mode Cluster

dbdev — Mon, 25 Aug 2025 15:58:40 GMT

Hi!

As the title suggests, I want to install Maven libaries on my cluster with access mode 'Standard'. Our workspace is VNet injected and has Unity Catalog enabled.

The coordinates have been allowlisted by the account team according to these instructions: Allowlist libraries and init scripts on compute with standard access mode (formerly shared access mode) - Azure Databricks | Azure Docs

And in the Cluster UI it seems to have been installed succesfully:

But when I try to execute any code, I get the following error:

The driver logs indicate a successful installation as well:

So I don't know what I'm doing wrong. Without the library installed, the cluster can execute code fine.

Is there some additional whitelisting necessary? Any suggestions?

Thank you in advance 🙂

Re: Maven libraries in VNet injected, UC enabled workspace on Standard Access Mode Cluster

szymon_dybczak — Mon, 25 Aug 2025 17:18:24 GMT

Hi @dbdev ,

Could you provide your DBR runtime version and full specification of a cluster? This is most often caused by incompatibility of this particular library version with DBR runtime

Re: Maven libraries in VNet injected, UC enabled workspace on Standard Access Mode Cluster

nayan_wylde — Mon, 25 Aug 2025 20:10:59 GMT

@dbdev Are you using any any artifactory or any private repo to store the mavens. Currently with standard and dedicated mode we have found this issue that maven packages stored in artifactory and private repo is not supported. So we have loaded the jar file in volumes and installing the package from volumes.

Re: Maven libraries in VNet injected, UC enabled workspace on Standard Access Mode Cluster

dbdev — Tue, 26 Aug 2025 07:10:47 GMT

@nayan_wylde We're just using the Maven default repository, pre-configured by Databricks.

@szymon_dybczak Sure! DBR is 15.4, but I've tried with 16.4 as well.

In another workspace that doesn't have unity catalog enabled, I was able to install and use the Maven library with no issues. The cluster configuration was the same, only access mode was different: set to 'No isolation shared'.

I tried the 'No isolation shared' access mode as well in my UC workspace, but still the same error.

Re: Maven libraries in VNet injected, UC enabled workspace on Standard Access Mode Cluster

szymon_dybczak — Tue, 26 Aug 2025 07:27:15 GMT

Great, thanks for additional details @dbdev . Could you also provide specific version of oracle jdbc library that you're trying to install? I'll try to recreate your scenario on my evironemnt
So, please provide maven coordinates 🙂

BTW, could you try to use dedicated access mode in the meantime?

Re: Maven libraries in VNet injected, UC enabled workspace on Standard Access Mode Cluster

dbdev — Tue, 26 Aug 2025 07:50:09 GMT

com.oracle.database.jdbc:ojdbc8:23.8.0.25.04

I just tried with dedicated: same error.

Thank you for the replies and help 😊

Re: Maven libraries in VNet injected, UC enabled workspace on Standard Access Mode Cluster

dbdev — Mon, 01 Sep 2025 07:19:19 GMT

@nayan_wylde @szymon_dybczak I just tried using a JAR I uploaded to an allowlisted Volume (ojdbc8 of oracle) and I get the same error. it seems like I'm able to install an JAR, but once it's installed my cluster is broken.

Re: Maven libraries in VNet injected, UC enabled workspace on Standard Access Mode Cluster

szymon_dybczak — Mon, 01 Sep 2025 08:34:18 GMT

Hi @dbdev ,

Sorry, I'm completely forgot to reply. So I've created UC enabled cluster with the same runtime as yours within VNET Injected workspace and I've installed this library.

Installation went smoothly and I'm not getting any error with cluster:

Could you check your drivers logs? Maybe there we can find something useful. And do you have some other libraries installed except com.oracle.database.jdbc:ojdbc8:23.8.0.25.04?

Re: Maven libraries in VNet injected, UC enabled workspace on Standard Access Mode Cluster

dbdev — Mon, 01 Sep 2025 09:43:02 GMT

In the driverlogs we get errors saying that our clusters can't connect to the hive metastore mysql database.
We're not planning to use the hive metastore, but these errors make it difficult to debug this issue as it results in a lot of error output, so we're trying to resolve it. I'll update once we have resolved these :).

Sidenote, do you think this metastore connection issue could cause this JAR issue? If I don't install JARs/Maven libs, I can run code on my cluster just fine, even while still getting this metastore connection issue, so I didn't think much of it, as we're not planning to use the metastore.

Re: Maven libraries in VNet injected, UC enabled workspace on Standard Access Mode Cluster

dbdev — Thu, 04 Sep 2025 16:47:32 GMT

We have resolved the Metastore issue, which also seemed to have resolved the JAR issue. I don't have a clue why this resolves it. The network people might have used service tags which also opened the workspace to the odbc connections?

Re: Maven libraries in VNet injected, UC enabled workspace on Standard Access Mode Cluster

szymon_dybczak — Thu, 04 Sep 2025 17:02:01 GMT

Thanks for sharing this. I wouldn't have thought that lack of hive metastore connection could cause issue with installing jars.