cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to execute a spark-submit on a databricks job for a spring-boot jar built using maven. Failing with error : Error: Failed to load class com.****.settlement.jobs.EntryPoint.

DineshNO
New Contributor II

I have setup a spring boot application which works as expected as a standalone spring boot app.

When i build the jar and try to set it up as a databricks job, i am facing these issues.

i am getting same error in local as well.

I have tried using maven-shade-plugin to make it compatible with databricks as i read in some post.

/usr/local/Cellar/apache-spark/3.2.1 ./bin/spark-submit \
  --class com.****.settlement.jobs.EntryPoint \
  --master local \
  /Users/dineshkumar/.m2/repository/com/****/settlement/jobs/exchange-rate-jobs/0.0.1-SNAPSHOT/exchange-rate-jobs-0.0.1-SNAPSHOT.jar \
  1000

22/04/29 12:29:04 WARN Utils: Your hostname, dineshkumar-macbookpro.local resolves to a loopback address: 127.0.0.1; using 192.168.225.161 instead (on interface en0)

22/04/29 12:29:04 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/3.2.1/libexec/jars/spark-unsafe_2.12-3.2.1.jar) to constructor java.nio.DirectByteBuffer(long,int)

WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

Error: Failed to load class com.tidal.settlement.jobs.EntryPoint.

log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

1 REPLY 1

Atanu
Esteemed Contributor
Esteemed Contributor

could you please try with python terminal and see how that behaves?I am not 100% sure if this is relates to your use case.

@Dinesh L​ 

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.