how to remove empty rows from the data frame.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-20-2018 12:33 AM
lets assume if i have 10 columns in a data frame,all 10 columns has empty values for 100 rows out of 200 rows, how i can skip the empty rows?
- Labels:
-
Spark sql
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-20-2018 08:58 AM
You can use the .drop() function, which is available as part of DataFrameNaFunctions.
With Scala:
https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/DataFrameNaFunctions.html#drop()
With Python:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-20-2018 11:47 AM
@dillon,
Drop will not help me in my case , i have a 200 rows in a table which has 100 rows as null and rest of rows has proper values .
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-15-2019 07:33 AM
Simply you can just use the .drop() function, which is available as part of DataFrameNaFunctions.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-26-2019 03:31 AM
@searro , it would be nice to have a well-formed example.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-09-2019 04:10 AM
How can we do this in Java.
I have a DataFrame with three columns, When all three values are null. Is there a way to delete that row in java?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-18-2020 10:44 AM
you can try this: df.na.drop(how = "all"), this will remove the row only if all the rows are null or NaN
