topic Re: Does Databricks have a maven repository to download the jars? in Data Engineering

Does Databricks have a maven repository to download the jars?

brickster_2018 — Thu, 24 Jun 2021 06:39:07 GMT

Using OSS jars is causing classpath issues always when running the job on Databricks. The same job works fine on EMR/on-premise.

Re: Does Databricks have a maven repository to download the jars?

brickster_2018 — Thu, 24 Jun 2021 08:33:02 GMT

Databricks does not host the jars in its own maven repository. The jars from OSS can be used to compile the application. The OSS jars should not be used at the execution time. ie: the application jars should not be fat jars, but rather thin jars.

If internal Spark APIs are used, then it's possible the Databricks version of those classes has a different method, and compiling your application against the OSS jars causes Classpath issues. In such scenarios the jars available on the cluster should be sourced from your local and used for compilation. The jars are available at :

https://docs.databricks.com/dev-tools/databricks-connect.html#intellij-scala-or-java

Re: Does Databricks have a maven repository to download the jars?

mj2022 — Wed, 27 Jul 2022 17:10:10 GMT

I following the https://docs.databricks.com/dev-tools/databricks-connect.html#intellij-scala-or-java to obtain spark-avro jar since databricks have it's custom from_avro method to use with kafka schema registry, But i am not able to find spark-avro jar using databricks connect ..We need that jar to be used with compilation. Any ideas why spark-avro jar is missing when we get jars using "databricks-connect get-jar-dir"?