Py4JJavaError: An error occurred while calling o465.coun

[sinclair]

The following error occured when running .count() on a big sparkDF. 

Py4JJavaError: An error occurred while calling o465.count. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 6 in stage 3.0 failed 4 times, most recent failure: Lost task 6.3 in stage 3.0 (TID 32) (10.139.64.6 executor 0): java.lang.NullPointerException

Data has close to 1.2mil records and have some nested json objects, so it's pretty big. I am using  cluster with 64GB memory and 16GB cores and 16gb worker with 4gb cores. 

Is there a way to solve this count? Is increasing cluster size the only solution?

[iakshaykr]

DBR : ? 
Number of workers: ? 
number of driver node : assuming 1 driver

[sinclair]

DBR: 13.3

Min 2 and max 8 workers

Driver node is 1 

[p4pratikjain]

can you provide complete stacktrace ?

[iakshaykr]

@sinclair - Can you please use num_workers=2 or 5 which will be constant instead of using auto scaling ? 

Meantime i am trying to reproduce the same issue in my end ! 

[szymon_dybczak]

Hi @sinclair,

Maybe try to increase number of partitions. There are no enough details, but assuming this error occur on worker due to insufficient memory, increasing partition number can help

[Rishabh_Tiwari]

Hi @sinclair ,

Thank you for reaching out to our community! We're here to help you. 

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.

If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!

Thanks,

Rishabh