Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-13-2023 09:36 AM
@Saswata Dutta Welcome to the club. Wish you a great time with Spark.
The filter API always need the parameters in parenthesis for equality checks.
//Filter multiple condition
df.filter( (df.state == "OH") & (df.gender == "M") ) \
.show(truncate=False) In your case, you missed the bracket in the condition. The below code should work.
df.filter((F.col("state").isNull())| (F.col("state")=="")| (F.col("state").contains(""))| (F.col("number").isNull())).show()Please try and see if this helps.