Hi @Nagini Sitaraman​ 

Can you try forcing the GC to unload the objects in the driver's memory. That should solve your issue.

code to do that :

spark.catalog.clearCache()

for (id, rdd) in spark.sparkContext._jsc.getPersistentRDDs().items():

  rdd.unpersist()

  print("Unloaded {} rdd".format(id))

Cheers