topic Re: How to install a JAR library via a global init script? in Data Engineering

How to install a JAR library via a global init script?

ncouture — Mon, 14 Nov 2022 02:10:08 GMT

I have a JAR I want to be installed as a library on all clusters. I have tried both

wget /databricks/jars/ some_repo

and

cp /dbfs/FileStore/jars/name_of_jar.jar /databricks/jars/

clusters start up but the JAR is not installed as a library. I am aware that it might not show in the UI under libraries however its effect allows the querying of JSON formatted tables. Testing with a newly started/created table results in an error when querying such a table. The only way I have gotten this to work is through the GUI on a cluster by cluster basis using the path in the second snippet. So the file is good.

What am I missing to doing this programmatically via an init script?

Re: How to install a JAR library via a global init script?

karthik_p — Tue, 15 Nov 2022 18:27:01 GMT

@Nicholas Couture please try below steps in article, if you have not come across, you need to go with global init config please. Cluster node initialization scripts | Databricks on AWS

Re: How to install a JAR library via a global init script?

ncouture — Tue, 15 Nov 2022 22:13:15 GMT

Yes I have viewed that article already. It contains nothing new or helpful.

I can see the JAR in the system classpath of the Spark UI but it doesn't seem to be installed/initialised for the cluster. This is really quite annoying

Re: How to install a JAR library via a global init script?

ncouture — Fri, 09 Dec 2022 04:30:45 GMT

Found a solution.

echo /databricks/databricks-hive /databricks/jars /databricks/glue | xargs -n 1 cp /dbfs/FileStore/jars/NAME_OF_THE_JAR.jar

had to first add the jar as a library through the GUI via Create -> Library then uploaded the downloaded JAR. Copying it into the three locations noted sorted it out

Re: How to install a JAR library via a global init script?

EliCunningham — Tue, 20 May 2025 06:26:51 GMT

Ensure your init script installs the JAR correctly on cluster startup.