How do I download maven libraries with dependencies?

User16871418122 — Fri, 07 May 2021 14:28:10 GMT

I want to import a maven library with its dependencies. How to do it?

Re: How do I download maven libraries with dependencies?

User16871418122 — Fri, 07 May 2021 14:43:06 GMT

I recommend creating a UBER jar or download jars offline use it in clusters when the maven becomes healthy again:

1. Install the MVN CLI tool on your local mac:

 brew install mvnvm

2. Download the Artifact with all dependencies:

mvn dependency:get -DrepoUrl=http://packages.confluent.io/maven/ -DgroupId=org.apache.kafka -DartifactId=kafka_2.11 -Dversion=0.10.0.0-cp1

mvn dependency:get -DrepoUrl=http://packages.confluent.io/maven/ -DgroupId=org.apache.kafka -DartifactId=kafka_2.11 -Dversion=0.10.0.0-cp1

3. Change directory into maven download repository and find the path of downloaded local path of repository:

cd $HOME/.m2

find ./ -name kafka-clients-0.10.0.0-cp1*

4. Copy the jar and dependencies to location /Users/user-gbth/mvn-lib-kafka/

mkdir /Users/user-gbth/mvn-lib-kafka/

cp /Users/user-gbth/.m2/repository/org/apache/kafka/kafka-clients/0.10.0.0-cp1/kafka-clients-0.10.0.0-cp1.jar /Users/gobinath/mvn-lib-kafka/

mvn dependency:copy-dependencies -f /Users/user-gbth/.m2/repository/org/apache/kafka/kafka-clients/0.10.0.0-cp1/kafka-clients-0.10.0.0-cp1.pom -DoutputDirectory=/Users/user-gbth/mvn-lib-kafka/

mvn dependency:copy-dependencies -f /Users/user-gbth/.m2/repository/org/apache/kafka/kafka-clients/0.10.0.0-cp1/kafka-clients-0.10.0.0-cp1.pom -DoutputDirectory=/Users/user-gbth/mvn-lib-kafka/

5. Zip the file and import the zip file into Databricks.

zip -r kafka-clients-0.10.0.0-cp1.zip /Users/gobinath/mvn-lib-kafka/

You can upload via UI or upload to the s3 bucket and load into databricks.

6) Load jars in databricks:

a) extract the zip to /databricks/jar location using databricks init script:

#!/bin/bash
unzip /dbfs/path/kafka-clients-0.10.0.0-cp1.zip –d /databricks/jars/

b) or wrap it as UBER jar and attach to the library dependency list of the cluster.

https://maven.apache.org/plugins/maven-shade-plugin/examples/includes-excludes.html

topic Re: How do I download maven libraries with dependencies? in Data Engineering

How do I download maven libraries with dependencies?

Re: How do I download maven libraries with dependencies?