topic Re: how to remove empty rows from the data frame. in Data Engineering

how to remove empty rows from the data frame.

dhanunjaya — Thu, 20 Sep 2018 07:33:32 GMT

lets assume if i have 10 columns in a data frame,all 10 columns has empty values for 100 rows out of 200 rows, how i can skip the empty rows?

dillon_bostwick — Thu, 20 Sep 2018 15:58:13 GMT

You can use the .drop() function, which is available as part of DataFrameNaFunctions.

With Scala:

With Python:

dhanunjaya — Thu, 20 Sep 2018 18:47:15 GMT

@dillon,

Drop will not help me in my case , i have a 200 rows in a table which has 100 rows as null and rest of rows has proper values .

searro — Sun, 15 Sep 2019 14:33:04 GMT

Simply you can just use the .drop() function, which is available as part of DataFrameNaFunctions.

Malan_Francois_ — Thu, 26 Sep 2019 10:31:22 GMT

@searro , it would be nice to have a well-formed example.

bhuvanrjb — Mon, 09 Dec 2019 12:10:23 GMT

How can we do this in Java.

I have a DataFrame with three columns, When all three values are null. Is there a way to delete that row in java?

GaryDiaz — Wed, 18 Mar 2020 17:44:03 GMT

you can try this: df.na.drop(how = "all"), this will remove the row only if all the rows are null or NaN