# Set the chunk size in bytes (1MB = 1048576 bytes)
chunk_size = 1048576
 
# Fix the number of partitions needed based on the chunk size
num_partitions = (df.rdd.map(lambda x: len(str(x))).sum() // chunk_size) + 1
 
# Repartition the DataFrame using maxRecordsPerFile option
df_chunks = df.repartition(num_partitions).write.option("maxRecordsPerFile", chunk_size).mode("overwrite").parquet("/output_path")

you can split the dataframe using the method above.

Once you have the partitions, traverse across each partition and collect the rows indiviudally. you can then add up these rows for each partition.

#DAIS2023