cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Resolving NoClassDefFoundError: org/apache/spark/Logging exception

Mahesha999
New Contributor II

I was trying out hbase-spark connector. To start with, I am trying out this code. My pom dependencies are:

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.0.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.0.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-spark</artifactId>
        <version>2.0.0-alpha4</version>
    </dependency>
</dependencies> 

I am getting following exception while running the code: Exception in thread "main"

java.lang.NoClassDefFoundError: org/apache/spark/Logging
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.hadoop.hbase.spark.JavaHBaseContext.<init>(JavaHBaseContext.scala:46)
    at com.myproj.poc.sparkhbaseneo4j.App.main(App.java:71)Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 14 more 

com.myproj.poc.sparkhbaseneo4j.App.main(App.java:71)
is line 67 in github code.

I checked this thread. It says that I should include same versions of all libraries. Earlier, I had 2.3.0 versions of spark libraries in my pom. But I realized that

hbase-spark
has latest version
2.0.0
. So I downgraded versions of of all spark libraries to
2.0.0
. But I am still getting the same exception.

Or do I have to stick to

1.X.X
versions only for using this, as this answer says its been removed after version 1.5.2?

3 REPLIES 3

User16301467518
New Contributor II

The alpha of hbase-spark you're using depends on Spark 1.6 -- see hbase-spark/pom.xml:L33 -- so you'll probably have to stick with 1.6 if you want to use that published jar.

For reasons I don't understand hbase-spark was removed in the last couple of HBase RCs but it's now in master with Spark 2.1.x dependencies. You could go ahead and build from source and try that.

Got rid of exception by adding following dependency:

<dependency>
    <groupId>commons-logging</groupId>
    <artifactId>commons-logging</artifactId>
    <version>1.1.1</version>
</dependency>

but I dont know whether its right way to resolve it. Is it?

It'd help to see the stack trace and SLF4J output, here or as a gist. Before adding commons-logging it's possible SLF4J failed to get a logger and then

logX()
was called triggering the NoClassDefFoundError.

I don't know if the 1.6-based code will run correctly and since these jars are from an alpha, I'm wary.

HBase 2.0.0 was released last week, but hbase-spark was held back. If you build from master it's there.

One alternative worth considering is using Apache Phoenix. It often performs better and would tightly align with DataFrames. It appears that since 4.10 it uses Spark 2.0.x.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group