How to execute a spark-submit on a databricks job for a spring-boot jar built using maven. Failing with error : Error: Failed to load class com.****.settlement.jobs.EntryPoint.

DineshNO — Fri, 29 Apr 2022 07:15:17 GMT

I have setup a spring boot application which works as expected as a standalone spring boot app.

When i build the jar and try to set it up as a databricks job, i am facing these issues.

i am getting same error in local as well.

I have tried using maven-shade-plugin to make it compatible with databricks as i read in some post.

/usr/local/Cellar/apache-spark/3.2.1 ./bin/spark-submit \
  --class com.****.settlement.jobs.EntryPoint \
  --master local \
  /Users/dineshkumar/.m2/repository/com/****/settlement/jobs/exchange-rate-jobs/0.0.1-SNAPSHOT/exchange-rate-jobs-0.0.1-SNAPSHOT.jar \
  1000

22/04/29 12:29:04 WARN Utils: Your hostname, dineshkumar-macbookpro.local resolves to a loopback address: 127.0.0.1; using 192.168.225.161 instead (on interface en0)

22/04/29 12:29:04 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/3.2.1/libexec/jars/spark-unsafe_2.12-3.2.1.jar) to constructor java.nio.DirectByteBuffer(long,int)

WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

Error: Failed to load class com.tidal.settlement.jobs.EntryPoint.

log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Re: How to execute a spark-submit on a databricks job for a spring-boot jar built using maven. Failing with error : Error: Failed to load class com.****.settlement.jobs.EntryPoint.

Atanu — Tue, 13 Sep 2022 18:34:45 GMT

could you please try with python terminal and see how that behaves?I am not 100% sure if this is relates to your use case.

@Dinesh L

topic How to execute a spark-submit on a databricks job for a spring-boot jar built using maven. Failing with error : Error: Failed to load class com.****.settlement.jobs.EntryPoint. in Data Engineering

How to execute a spark-submit on a databricks job for a spring-boot jar built using maven. Failing with error : Error: Failed to load class com.****.settlement.jobs.EntryPoint.

Re: How to execute a spark-submit on a databricks job for a spring-boot jar built using maven. Failing with error : Error: Failed to load class com.****.settlement.jobs.EntryPoint.