Confusion in string comparison

[Sas]

Hello expert

I am new to spark. I am using same price of code but getting different results

When i am using below piece of code, i am getting error

py4j.Py4JException: Method or([class java.lang.String]) does not exist

df.filter(F.col("state").isNull()

     | F.col("state")==""

     | F.col("state").contains("")

     | F.col("number").isNull()).show()

However when i am using below piece of code, its working fine

df.withColumn("state",

       F.when(F.col("state")=="",None)

       .otherwise(F.col("state"))).show()

The same F.col("state")=="" code is working in one place but not working in other

[Ajay-Pandey]

Hi @Saswata Dutta​ ,

Please use blow code this will work-

df.filter((F.col("state").isNull())| (F.col("state")=="")| (F.col("state").contains(""))| (F.col("number").isNull())).show()

[pvignesh92 · Accepted solution]

@Saswata Dutta​ Welcome to the club. Wish you a great time with Spark.

The filter API always need the parameters in parenthesis for equality checks.

//Filter multiple condition
df.filter( (df.state  == "OH") & (df.gender  == "M") ) \
    .show(truncate=False)  

In your case, you missed the bracket in the condition. The below code should work.

df.filter((F.col("state").isNull())| (F.col("state")=="")| (F.col("state").contains(""))| (F.col("number").isNull())).show()

Please try and see if this helps.

[Anonymous]

Hi @Saswata Dutta​ 

Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.

Please help us select the best solution by clicking on "Select As Best" if it does.

Your feedback will help us ensure that we are providing the best possible service to you.

Thank you!