Classpath issues when running spark-submit

brickster_2018
Databricks Employee
Databricks Employee

How to identify the jars used to load a particular class. I am sure I packed the classes correctly in my application jar. However, looks like the class is loaded from a different jar. I want to understand the details so that I can ensure to use the right version of the jar and avoid conflict.

brickster_2018
Databricks Employee
Databricks Employee

Adding the below configurations at the cluster level can help to print more logs to identify the jars from which the class is loaded.

  • spark.executor.extraJavaOptions=-verbose:class
  • spark.driver.extraJavaOptions=-verbose:class

View solution in original post