cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to install a JAR library via a global init script?

ncouture
Contributor

I have a JAR I want to be installed as a library on all clusters. I have tried both

wget /databricks/jars/ some_repo

and

cp /dbfs/FileStore/jars/name_of_jar.jar /databricks/jars/

clusters start up but the JAR is not installed as a library. I am aware that it might not show in the UI under libraries however its effect allows the querying of JSON formatted tables. Testing with a newly started/created table results in an error when querying such a table. The only way I have gotten this to work is through the GUI on a cluster by cluster basis using the path in the second snippet. So the file is good.

What am I missing to doing this programmatically via an init script?

1 ACCEPTED SOLUTION

Accepted Solutions

ncouture
Contributor

Found a solution.

echo /databricks/databricks-hive /databricks/jars /databricks/glue | xargs -n 1 cp /dbfs/FileStore/jars/NAME_OF_THE_JAR.jar

had to first add the jar as a library through the GUI via Create -> Library then uploaded the downloaded JAR. Copying it into the three locations noted sorted it out

View solution in original post

3 REPLIES 3

karthik_p
Esteemed Contributor

@Nicholas Couture​ please try below steps in article, if you have not come across, you need to go with global init config please. Cluster node initialization scripts | Databricks on AWS

Yes I have viewed that article already. It contains nothing new or helpful.

I can see the JAR in the system classpath of the Spark UI but it doesn't seem to be installed/initialised for the cluster. This is really quite annoying

ncouture
Contributor

Found a solution.

echo /databricks/databricks-hive /databricks/jars /databricks/glue | xargs -n 1 cp /dbfs/FileStore/jars/NAME_OF_THE_JAR.jar

had to first add the jar as a library through the GUI via Create -> Library then uploaded the downloaded JAR. Copying it into the three locations noted sorted it out

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!