topic Re: How to see if condition is True / False for all rows in a DataFrame? in Data Engineering

How to see if condition is True / False for all rows in a DataFrame?

Mado — Tue, 20 Dec 2022 09:01:42 GMT

Assume that I have a Spark DataFrame, and I want to see if records satisfy a condition.

Example dataset:

# Prepare Data
data = [('A', 1), \
        ('A', 2), \
        ('B', 3)
  ]
 
# Create DataFrame
columns= ['col_1', 'col_2']
df = spark.createDataFrame(data = data, schema = columns)
df.show(truncate=False)

If I run the following code in Databricks:

In the output, I don't see if condition is met. If I create a pandas DataFrame:

import pandas as pd
pdf = pd.DataFrame(data, columns=columns)

I can check if condition is met for all rows:

How can I get the same output when working with Spark DataFrame?

Re: How to see if condition is True / False for all rows in a DataFrame?

UmaMahesh1 — Tue, 20 Dec 2022 09:21:28 GMT

Hi @Mohammad Saber

Since your output will be a column object you just need to use df for that.

The following will work.

df.select(df['col_1'] == 'A').show()

Hope this helps... Do mark it as an answer if it helps..

Cheers

Re: How to see if condition is True / False for all rows in a DataFrame?

Ajay-Pandey — Tue, 20 Dec 2022 11:57:33 GMT

Hi you can use display() or show() function that will provide you expected results.

Re: How to see if condition is True / False for all rows in a DataFrame?

Mado — Tue, 20 Dec 2022 12:12:58 GMT

Thanks. How can I change the column name "(col_1=A)" to e.g. "Condition"?

Re: How to see if condition is True / False for all rows in a DataFrame?

UmaMahesh1 — Tue, 20 Dec 2022 12:23:20 GMT

You can use a "when otherwise" and give the condition you want. It works similar to sql case when query.

E.g.

df.select(when(df['col_1'] == 'A', "Condition1").otherwise("Condition2")).show()

Re: How to see if condition is True / False for all rows in a DataFrame?

Mado — Tue, 20 Dec 2022 12:26:49 GMT

Thanks. I meant changing the column name. I tried:

df.select( df['col_1']=='A').alias('Condition').show()

But, it didn't work.

Re: How to see if condition is True / False for all rows in a DataFrame?

UmaMahesh1 — Tue, 20 Dec 2022 12:30:17 GMT

That alias needs to wrapped inside the select function to rename the column.

df.select((df['col_1'] == 'A').alias("Condition")).show()