- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-12-2023 05:49 PM
Hello expert
I am new to spark. I am using same price of code but getting different results
When i am using below piece of code, i am getting error
py4j.Py4JException: Method or([class java.lang.String]) does not exist
df.filter(F.col("state").isNull()
| F.col("state")==""
| F.col("state").contains("")
| F.col("number").isNull()).show()
However when i am using below piece of code, its working fine
df.withColumn("state",
F.when(F.col("state")=="",None)
.otherwise(F.col("state"))).show()
The same F.col("state")=="" code is working in one place but not working in other
- Labels:
-
Different Results
-
String
-
String Comparison
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-13-2023 09:36 AM
@Saswata Dutta Welcome to the club. Wish you a great time with Spark.
The filter API always need the parameters in parenthesis for equality checks.
//Filter multiple condition
df.filter( (df.state == "OH") & (df.gender == "M") ) \
.show(truncate=False)
In your case, you missed the bracket in the condition. The below code should work.
df.filter((F.col("state").isNull())| (F.col("state")=="")| (F.col("state").contains(""))| (F.col("number").isNull())).show()
Please try and see if this helps.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-13-2023 06:14 AM
Hi @Saswata Dutta ,
Please use blow code this will work-
df.filter((F.col("state").isNull())| (F.col("state")=="")| (F.col("state").contains(""))| (F.col("number").isNull())).show()
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-13-2023 09:36 AM
@Saswata Dutta Welcome to the club. Wish you a great time with Spark.
The filter API always need the parameters in parenthesis for equality checks.
//Filter multiple condition
df.filter( (df.state == "OH") & (df.gender == "M") ) \
.show(truncate=False)
In your case, you missed the bracket in the condition. The below code should work.
df.filter((F.col("state").isNull())| (F.col("state")=="")| (F.col("state").contains(""))| (F.col("number").isNull())).show()
Please try and see if this helps.
![](/skins/images/1C7D039E274DA4E433FB1B1A3EAE173A/responsive_peak/images/icon_anonymous_profile.png)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-18-2023 12:35 AM
Hi @Saswata Dutta
Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.
Please help us select the best solution by clicking on "Select As Best" if it does.
Your feedback will help us ensure that we are providing the best possible service to you.
Thank you!
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)