How to find the size or shape of a DataFrame in PySpark?

[Ryan_Chynoweth · Accepted solution]

Assume that "df" is a Dataframe. The following code (with comments) will show various options to describe a dataframe.

# get a row count
df.count()
 
# get the approximate count (faster than the .count())
df.rdd.countApprox()
 
# print the schema (shape of your df)
df.printSchema()
 
# get the columns as a list
df.columns
 
# get the columns and types as tuples in a list
df.dtypes

[Kaniz]

Thank you @Ryan Chynoweth​  for your answer.

[Feras]

To obtain the shape of a data frame in PySpark, you can obtain the number of rows through "DF.count()" and the number of columns through "len(DF.columns)". The code below provides the shape of PySpark data frame "DF".

print((DF.count(), len(DF.columns)))

[Kaniz]

Thank you @Feras Seder​ for the answer.