how to remove empty rows from the data frame.

[dhanunjaya]

lets assume if i have 10 columns in a data frame,all 10 columns has empty values for 100 rows out of 200 rows, how i can skip the empty rows?

[dillon_bostwick]

You can use the .drop() function, which is available as part of DataFrameNaFunctions.

With Scala:

https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/DataFrameNaFunctions.html#drop()

With Python:

https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=nafunctions#pyspark.sql.D...

[dhanunjaya]

@dillon,

Drop will not help me in my case , i have a 200 rows in a table which has 100 rows as null and rest of rows has proper values .

[searro]

Simply you can just use the .drop() function, which is available as part of DataFrameNaFunctions.

[Malan_Francois_]

@searro , it would be nice to have a well-formed example.

[bhuvanrjb]

How can we do this in Java.

I have a DataFrame with three columns, When all three values are null. Is there a way to delete that row in java?

[GaryDiaz]

you can try this: df.na.drop(how = "all"), this will remove the row only if all the rows are null or NaN