topic Re: Confusion in string comparison in Data Engineering

Confusion in string comparison

Sas — Mon, 13 Mar 2023 00:49:40 GMT

Hello expert

I am new to spark. I am using same price of code but getting different results

When i am using below piece of code, i am getting error

py4j.Py4JException: Method or([class java.lang.String]) does not exist

df.filter(F.col("state").isNull()

| F.col("state")==""

| F.col("state").contains("")

| F.col("number").isNull()).show()

However when i am using below piece of code, its working fine

df.withColumn("state",

F.when(F.col("state")=="",None)

.otherwise(F.col("state"))).show()

The same F.col("state")=="" code is working in one place but not working in other

Ajay-Pandey — Mon, 13 Mar 2023 13:14:30 GMT

Hi @Saswata Dutta ,

Please use blow code this will work-

df.filter((F.col("state").isNull())| (F.col("state")=="")| (F.col("state").contains(""))| (F.col("number").isNull())).show()

pvignesh92 — Mon, 13 Mar 2023 16:36:48 GMT

@Saswata Dutta Welcome to the club. Wish you a great time with Spark.

The filter API always need the parameters in parenthesis for equality checks.

//Filter multiple condition
df.filter( (df.state  == "OH") & (df.gender  == "M") ) \
    .show(truncate=False)

In your case, you missed the bracket in the condition. The below code should work.

df.filter((F.col("state").isNull())| (F.col("state")=="")| (F.col("state").contains(""))| (F.col("number").isNull())).show()

Please try and see if this helps.

Anonymous — Sat, 18 Mar 2023 07:35:10 GMT

Hi @Saswata Dutta

Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.

Please help us select the best solution by clicking on "Select As Best" if it does.

Your feedback will help us ensure that we are providing the best possible service to you.

Thank you!