topic Re: Py4JJavaError: An error occurred while calling o465.coun in Data Engineering

Py4JJavaError: An error occurred while calling o465.coun

sinclair — Sat, 13 Jul 2024 03:48:23 GMT

The following error occured when running .count() on a big sparkDF.

Py4JJavaError: An error occurred while calling o465.count. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 6 in stage 3.0 failed 4 times, most recent failure: Lost task 6.3 in stage 3.0 (TID 32) (10.139.64.6 executor 0): java.lang.NullPointerException

Data has close to 1.2mil records and have some nested json objects, so it's pretty big. I am using cluster with 64GB memory and 16GB cores and 16gb worker with 4gb cores.

Is there a way to solve this count? Is increasing cluster size the only solution?

Re: Py4JJavaError: An error occurred while calling o465.coun

iakshaykr — Sat, 13 Jul 2024 20:11:05 GMT

DBR : ?
Number of workers: ?
number of driver node : assuming 1 driver

Re: Py4JJavaError: An error occurred while calling o465.coun

sinclair — Sat, 13 Jul 2024 21:50:32 GMT

DBR: 13.3

Min 2 and max 8 workers

Driver node is 1

Re: Py4JJavaError: An error occurred while calling o465.coun

p4pratikjain — Sun, 14 Jul 2024 07:53:29 GMT

can you provide complete stacktrace ?

Re: Py4JJavaError: An error occurred while calling o465.coun

iakshaykr — Sun, 14 Jul 2024 12:53:54 GMT

@sinclair - Can you please use num_workers=2 or 5 which will be constant instead of using auto scaling ?

Meantime i am trying to reproduce the same issue in my end !

Re: Py4JJavaError: An error occurred while calling o465.coun

szymon_dybczak — Sun, 14 Jul 2024 16:14:13 GMT

Hi @sinclair,

Maybe try to increase number of partitions. There are no enough details, but assuming this error occur on worker due to insufficient memory, increasing partition number can help

Re: Py4JJavaError: An error occurred while calling o465.coun

RishabhTiwari07 — Fri, 19 Jul 2024 12:56:49 GMT

Hi @sinclair ,

Thank you for reaching out to our community! We're here to help you.

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.

If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!

Thanks,

Rishabh