topic Re: Resolving NoClassDefFoundError: org/apache/spark/Logging exception in Data Engineering

Resolving NoClassDefFoundError: org/apache/spark/Logging exception

Mahesha999 — Fri, 27 Apr 2018 12:52:00 GMT

I was trying out hbase-spark connector. To start with, I am trying out this code. My pom dependencies are:

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.0.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.0.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-spark</artifactId>
        <version>2.0.0-alpha4</version>
    </dependency>
</dependencies>

I am getting following exception while running the code: Exception in thread "main"

java.lang.NoClassDefFoundError: org/apache/spark/Logging
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.hadoop.hbase.spark.JavaHBaseContext.<init>(JavaHBaseContext.scala:46)
    at com.myproj.poc.sparkhbaseneo4j.App.main(App.java:71)Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 14 more

com.myproj.poc.sparkhbaseneo4j.App.main(App.java:71)

is line 67 in github code.

I checked this thread. It says that I should include same versions of all libraries. Earlier, I had 2.3.0 versions of spark libraries in my pom. But I realized that

hbase-spark

has latest version

2.0.0

. So I downgraded versions of of all spark libraries to

2.0.0

. But I am still getting the same exception.

Or do I have to stick to

1.X.X

versions only for using this, as this answer says its been removed after version 1.5.2?

Re: Resolving NoClassDefFoundError: org/apache/spark/Logging exception

User16301467518 — Fri, 27 Apr 2018 14:36:40 GMT

The alpha of hbase-spark you're using depends on Spark 1.6 -- see hbase-spark/pom.xml:L33 -- so you'll probably have to stick with 1.6 if you want to use that published jar.

For reasons I don't understand hbase-spark was removed in the last couple of HBase RCs but it's now in master with Spark 2.1.x dependencies. You could go ahead and build from source and try that.

Re: Resolving NoClassDefFoundError: org/apache/spark/Logging exception

Mahesha999 — Thu, 03 May 2018 10:25:46 GMT

Got rid of exception by adding following dependency:

<dependency>
    <groupId>commons-logging</groupId>
    <artifactId>commons-logging</artifactId>
    <version>1.1.1</version>
</dependency>

but I dont know whether its right way to resolve it. Is it?

Re: Resolving NoClassDefFoundError: org/apache/spark/Logging exception

User16301467518 — Mon, 07 May 2018 14:36:19 GMT

It'd help to see the stack trace and SLF4J output, here or as a gist. Before adding commons-logging it's possible SLF4J failed to get a logger and then

logX()

was called triggering the NoClassDefFoundError.

I don't know if the 1.6-based code will run correctly and since these jars are from an alpha, I'm wary.

HBase 2.0.0 was released last week, but hbase-spark was held back. If you build from master it's there.

One alternative worth considering is using Apache Phoenix. It often performs better and would tightly align with DataFrames. It appears that since 4.10 it uses Spark 2.0.x.