07-27-2021 06:49 PM
Hi
I am getting "java.lang.NoClassDefFoundError: scala/Product$class" error while using Deequ 1.0.5 version. Please suggest fix to this problem or any work around
Error
Py4JJavaError Traceback (most recent call last) <command-2625366351750561> in <module>----> 1suggestionResult = ConstraintSuggestionRunner(spark)\ 2.onData(df)\ 3.addConstraintRule(DEFAULT())\ 4.run() 5 print(json.dumps(suggestionResult, indent=2))/databricks/python/lib/python3.8/site-packages/pydeequ/suggestions.py in addConstraintRule(self, constraintRule) 64for rule in constraintRule_jvm: 65 rule._set_jvm(self._jvm)---> 66rule_jvm = rule.rule_jvm 67 self._ConstraintSuggestionRunBuilder.addConstraintRule(rule_jvm) 68/databricks/python/lib/python3.8/site-packages/pydeequ/suggestions.py in rule_jvm(self) 184@property 185def rule_jvm(self):--> 186return self._deequSuggestions.rules.CategoricalRangeRule()
Code:
suggestionResult = ConstraintSuggestionRunner(spark) \ .onData(df) \ .addConstraintRule(DEFAULT()) \ .run() print(json.dumps(suggestionResult, indent=2))
Thanks
Ramabadran
09-02-2021 06:21 AM
Hi @ Ramabadran! My name is Kaniz, and I'm a technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers on the Forum have an answer to your questions first. Or else I will follow up shortly with a response.
01-07-2022 06:20 AM
I am having a similar issue and receiving this error. It appears that from other forums it is related to scala version. @Ramabadran were you ever able to figure it out? @Kaniz Fatma any response? Thank you.
01-07-2022 06:46 AM
Hi @David Foran and @Ramabadran , Which version are you using?
01-07-2022 06:49 AM
You’re running this version that isn’t compatible with databricks runtime version. Deequ 1.0.5 had a problem with Spark 3, so it’s better to use version 1.2.2 that is the latest.
11-29-2022 10:34 PM
Hi @Ramabadran ,
You need to go Cluster setting -> Libraries -> Install Now ->
Choose MAVEN ->com.databricks:spark-xml_2.12:0.15.0 Enter this text.
Boom Issue resolved!!!
12-01-2022 03:27 AM
its seems like maven issue
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!