topic Re: Could not initialize class error in Data Engineering

Could not initialize class error

Mohit_m — Mon, 27 Jun 2022 12:24:40 GMT

User is running a job triggered from ADF in Databricks. In this job they need to use custom libraries that are in jars. Most of the times jobs are running fine, however sometimes it fails with:

java.lang.NoClassDefFoundError: Could not initialize

Any suggestions?

Re: Could not initialize class error

Mohit_m — Mon, 27 Jun 2022 12:25:15 GMT

Can you please check if there are more than one jar containing this class . If multiple jars of the same type are available on the cluster, then there is no guarantee of JVM picking the proper classes for processing, which results in the intermittent behaviour of job run.

Re: Could not initialize class error

BillSchneider — Tue, 23 Apr 2024 14:37:10 GMT

I have a similar issue.

There is an intermittent issue with NoClassDefFoundError which looks like it's coming after several layers of ObjectInputStream.readObject -- perhaps trying to deserialize a reference in a shuffled object

The class appears once in a JAR, which is deployed once as a Library JAR for the job.

Most of the time it works fine, holding everything else constant, but there are occasional failures with this error. What could cause this failure to be intermittent and not consistent?