Spark code not running bcz of incorrect compute size

ashraf1395 — Wed, 10 Jul 2024 03:59:03 GMT

I have a dataset having 260 billion records

I need to group by 4 columns and find out the sum on four other columns

I increased the memory to e32 for driver and workers nodes, max workers is 40

The job still is stuck in this aggregate step where I’m writing it to the disk for persisting purpose.

Any solution?

We can use serverless compute but if I want to do it the normal way what can be the optimal size of cluster or some optimisations which I can do

Re: Spark code not running bcz of incorrect compute size

RishabhTiwari07 — Fri, 19 Jul 2024 11:37:56 GMT

Hi @ashraf1395 ,

Thank you for reaching out to our community! We're here to help you.

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.

If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!

Thanks,

Rishabh

topic Spark code not running bcz of incorrect compute size in Data Engineering

Spark code not running bcz of incorrect compute size

Re: Spark code not running bcz of incorrect compute size