Could not initialize class error

[Mohit_m]

User is running a job triggered from ADF in Databricks. In this job they need to use custom libraries that are in jars. Most of the times jobs are running fine, however sometimes it fails with:

java.lang.NoClassDefFoundError: Could not initialize

Any suggestions?

[Mohit_m · Accepted solution]

Can you please check if there are more than one jar containing this class . If multiple jars of the same type are available on the cluster, then there is no guarantee of JVM picking the proper classes for processing, which results in the intermittent behaviour of job run.

[BillSchneider]

I have a similar issue.

There is an intermittent issue with NoClassDefFoundError which looks like it's coming after several layers of ObjectInputStream.readObject -- perhaps trying to deserialize a reference in a shuffled object

The class appears once in a JAR, which is deployed once as a Library JAR for the job.

Most of the time it works fine, holding everything else constant, but there are occasional failures with this error. What could cause this failure to be intermittent and not consistent?