Databricks Community

dbdev · 3 weeks ago

Hi!

As the title suggests, I want to install Maven libaries on my cluster with access mode 'Standard'. Our workspace is VNet injected and has Unity Catalog enabled.

The coordinates have been allowlisted by the account team according to these instructions: Allowlist libraries and init scripts on compute with standard access mode (formerly shared access mo...

And in the Cluster UI it seems to have been installed succesfully:

But when I try to execute any code, I get the following error:

The driver logs indicate a successful installation as well:

So I don't know what I'm doing wrong. Without the library installed, the cluster can execute code fine.

Is there some additional whitelisting necessary? Any suggestions?

Thank you in advance 🙂

szymon_dybczak · 3 weeks ago

Hi @dbdev ,

Could you provide your DBR runtime version and full specification of a cluster? This is most often caused by incompatibility of this particular library version with DBR runtime

nayan_wylde · 3 weeks ago

@dbdev Are you using any any artifactory or any private repo to store the mavens. Currently with standard and dedicated mode we have found this issue that maven packages stored in artifactory and private repo is not supported. So we have loaded the jar file in volumes and installing the package from volumes.

dbdev · 3 weeks ago

@nayan_wylde We're just using the Maven default repository, pre-configured by Databricks.

@szymon_dybczak Sure! DBR is 15.4, but I've tried with 16.4 as well.

In another workspace that doesn't have unity catalog enabled, I was able to install and use the Maven library with no issues. The cluster configuration was the same, only access mode was different: set to 'No isolation shared'.

I tried the 'No isolation shared' access mode as well in my UC workspace, but still the same error.

szymon_dybczak · 3 weeks ago

Great, thanks for additional details @dbdev . Could you also provide specific version of oracle jdbc library that you're trying to install? I'll try to recreate your scenario on my evironemnt
So, please provide maven coordinates 🙂

BTW, could you try to use dedicated access mode in the meantime?

dbdev · 3 weeks ago

com.oracle.database.jdbc:ojdbc8:23.8.0.25.04

I just tried with dedicated: same error.

Thank you for the replies and help 😊

dbdev · 2 weeks ago

@nayan_wylde @szymon_dybczak I just tried using a JAR I uploaded to an allowlisted Volume (ojdbc8 of oracle) and I get the same error. it seems like I'm able to install an JAR, but once it's installed my cluster is broken.

szymon_dybczak · 2 weeks ago

Hi @dbdev ,

Sorry, I'm completely forgot to reply. So I've created UC enabled cluster with the same runtime as yours within VNET Injected workspace and I've installed this library.

Installation went smoothly and I'm not getting any error with cluster:

Could you check your drivers logs? Maybe there we can find something useful. And do you have some other libraries installed except com.oracle.database.jdbc:ojdbc8:23.8.0.25.04?

dbdev · 2 weeks ago

In the driverlogs we get errors saying that our clusters can't connect to the hive metastore mysql database.
We're not planning to use the hive metastore, but these errors make it difficult to debug this issue as it results in a lot of error output, so we're trying to resolve it. I'll update once we have resolved these :).

Sidenote, do you think this metastore connection issue could cause this JAR issue? If I don't install JARs/Maven libs, I can run code on my cluster just fine, even while still getting this metastore connection issue, so I didn't think much of it, as we're not planning to use the metastore.

dbdev · a week ago

We have resolved the Metastore issue, which also seemed to have resolved the JAR issue. I don't have a clue why this resolves it. The network people might have used service tags which also opened the workspace to the odbc connections?