Databricks Community

Ramabadran · ‎07-27-2021

Hi

I am getting "java.lang.NoClassDefFoundError: scala/Product$class" error while using Deequ 1.0.5 version. Please suggest fix to this problem or any work around

Error

Py4JJavaError Traceback (most recent call last) <command-2625366351750561> in <module>----> 1suggestionResult = ConstraintSuggestionRunner(spark)\ 2.onData(df)\ 3.addConstraintRule(DEFAULT())\ 4.run() 5 print(json.dumps(suggestionResult, indent=2))/databricks/python/lib/python3.8/site-packages/pydeequ/suggestions.py in addConstraintRule(self, constraintRule) 64for rule in constraintRule_jvm: 65 rule._set_jvm(self._jvm)---> 66rule_jvm = rule.rule_jvm 67 self._ConstraintSuggestionRunBuilder.addConstraintRule(rule_jvm) 68/databricks/python/lib/python3.8/site-packages/pydeequ/suggestions.py in rule_jvm(self) 184@property 185def rule_jvm(self):--> 186return self._deequSuggestions.rules.CategoricalRangeRule()

Code:

suggestionResult = ConstraintSuggestionRunner(spark) \ .onData(df) \ .addConstraintRule(DEFAULT()) \ .run() print(json.dumps(suggestionResult, indent=2))

Thanks

Ramabadran

Kaniz_Fatma · ‎09-02-2021

Hi @ Ramabadran! My name is Kaniz, and I'm a technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers on the Forum have an answer to your questions first. Or else I will follow up shortly with a response.

DaveForan · ‎01-07-2022

I am having a similar issue and receiving this error. It appears that from other forums it is related to scala version. @Ramabadran were you ever able to figure it out? @Kaniz Fatma any response? Thank you.

Kaniz_Fatma · ‎01-07-2022

Hi @David Foran and @Ramabadran , Which version are you using?

Kaniz_Fatma · ‎01-07-2022

You’re running this version that isn’t compatible with databricks runtime version. Deequ 1.0.5 had a problem with Spark 3, so it’s better to use version 1.2.2 that is the latest.