Aviral-Bhardwaj
Esteemed Contributor III

this will help you 

# Cluster config adjustments
spark.conf.set("spark.driver.memory", "16g") # Double current allocation
spark.conf.set("spark.driver.maxResultSize", "8g") # Prevent large collects

AviralBhardwaj