The following error occured when running .count() on a big sparkDF. 

Py4JJavaError: An error occurred while calling o465.count. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 6 in stage 3.0 failed 4 times, most recent failure: Lost task 6.3 in stage 3.0 (TID 32) (10.139.64.6 executor 0): java.lang.NullPointerException

Data has close to 1.2mil records and have some nested json objects, so it's pretty big. I am using  cluster with 64GB memory and 16GB cores and 16gb worker with 4gb cores. 

Is there a way to solve this count? Is increasing cluster size the only solution?