Classpath issues when running spark-submit

[brickster_2018]

How to identify the jars used to load a particular class. I am sure I packed the classes correctly in my application jar. However, looks like the class is loaded from a different jar. I want to understand the details so that I can ensure to use the right version of the jar and avoid conflict.

[brickster_2018 · Accepted solution]

Adding the below configurations at the cluster level can help to print more logs to identify the jars from which the class is loaded.

• spark.executor.extraJavaOptions=-verbose:class
• spark.driver.extraJavaOptions=-verbose:class