topic Re: how to check dataframe column value in Data Engineering

how to check dataframe column value

databicky — Tue, 17 Jan 2023 01:29:49 GMT

in my dataframe it have one column name like count, if that particular column value is greater than zero, the job needs to get failed, how can i perform that one?

Re: how to check dataframe column value

Chaitanya_Raju — Tue, 17 Jan 2023 03:26:19 GMT

Hi @Mohammed sadamusean

Can you try like below code in pyspark and let me know if you face any issues

variable_name = df.select(col("Column_Name")).collect()[0][0]
 
if(variable_name>0):
dbutils.notebook.exit('Notebook Failed')

Happy Learning!!

Re: how to check dataframe column value

Hubert-Dudek — Tue, 17 Jan 2023 09:44:03 GMT

Code without collect, which should not be used in production:

if df.filter("count > 0").count() > 0: dbutils.notebook.exit('Notebook Failed')

you can also use a more aggressive version:

if df.filter("count > 0").count() > 0: raise Exception("count bigger than 0")

Re: how to check dataframe column value

databicky — Tue, 17 Jan 2023 11:14:40 GMT

but it will get total count of the column right, but i need to check every specific column value

Re: how to check dataframe column value

Hubert-Dudek — Tue, 17 Jan 2023 12:02:50 GMT

first you filter for rows matching your query. You said that column is named count. Let's assume that column is called col instead, so filter("col > 0"), and then you apply the count() function, which will return how many rows match those criteria.

Re: how to check dataframe column value

databicky — Tue, 17 Jan 2023 12:17:33 GMT

it is working but how can we check the columns based on two values like count >0 and less than 0 , i tried with equal to 0 but it doesn't worked

Re: how to check dataframe column value

Hubert-Dudek — Tue, 17 Jan 2023 12:21:32 GMT

just put like in SQL

"colname > 0 OR colname< 0"

"colname != 0"