pvignesh92
Honored Contributor

@Saswata Dutta​ Welcome to the club. Wish you a great time with Spark.

The filter API always need the parameters in parenthesis for equality checks.

//Filter multiple condition
df.filter( (df.state  == "OH") & (df.gender  == "M") ) \
    .show(truncate=False)  

In your case, you missed the bracket in the condition. The below code should work.

df.filter((F.col("state").isNull())| (F.col("state")=="")| (F.col("state").contains(""))| (F.col("number").isNull())).show()

Please try and see if this helps.

View solution in original post