How to use from standalone Spark Jar running from Intellij Idea the library installed in Databricks DBR?

[Anonymous]

Hello,

I tried without success to use several libraries installed by use in the Databricks 9.1 cluster (not provived by default in DBR) from a standalone Spark application runs from Intellij Idea.

For instance, for connecting to Redshift it works only using the embedded Postgre driver.

If I try to use the Redshift driver and Sap Hana driver without success.

Libraries for have been installed as JARs in the Databricks cluster

• Sap Hana ngdbc_2_10_15.jar
• redshift_jdbc42_1_2_16_1027-330d8.jar

When I try to run in Debug mode or normal mode from Intellij Idea I get ClassnotFound.

Exception in thread "main" java.lang.ClassNotFoundException: com.sap.db.jdbc.Driver

SAP driver exists well in POM.XML dependencies, tried to add it to Intellij dependencies as well.

<dependency>

<groupId>com.sap.cloud.db.jdbc</groupId>

<artifactId>ngdbc</artifactId>

<version>2.10.15</version>

<type>jar</type>

</dependency>

Databricks connect JARS declared into Intellij dependencies properly: Basic Spark commands run well from Intellij standalone app to the cluster.

Technical environment:

• DBR 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12)
• Intellij Idea 2021.2.2 (from Windows 10)
• AWS Redshift driver or Sap Hana driver

Thank you four your support,

[Kaniz]

Hi @ vanausloos ! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[Anonymous]

Dear Kaniz,

Would you please provide an answer ? It is pretty urgent as we need to rewrite Databricks notebooks as standalone applications for better maintability.

Many thanks for your feedback.

[Anonymous · Accepted solution]

In addition to previous message, I am now able to run properly the job using Redshift driver from Databricks job: just add the Redshift library and the JAR for the job and it works.

However, not very convenient for debugging.

When I tried to run the app from Intellij to the remote Databricks cluster issue : Exception in thread "main" java.lang.ClassNotFoundException: Could not load an Amazon Redshift JDBC driver; see the README for instructions on downloading and configuring the official Amazon driver.

Tried :

• to add the driver JAR into Databricks connect jar path : no success
• use sc.addJars : no success
• use sc.addJars with dbfs path of the driver : no success

Is there a way to run in debug mode Scala app in Databricks cluster or not ?

Thank you

[Kaniz]

Hi @Xavier VAN AUSLOOS​ , This might help - https://databricks.com/blog/2016/10/18/7-tips-to-debug-apache-spark-code-faster-with-databricks.html

[Anonymous]

Dear Fatma

Sorry but no. It was not the question.

My customer (big German company will raise with Databricks support team directly). Please close this.

[Anonymous]

@Xavier VAN AUSLOOS​ - Would you be able to post the answer and mark it as best, so the community will be able to find the solution more easily if they run into this?

[Anonymous]

Unfortunately, I did not find any solution. We have to package JAR and run it from Databricks job for test/debug. Not efficient but as no solution for remote debug has been found/provided.