Fatal error when writing a big pandas dF

[chari]

Hello DB community,

I was trying to write a pandas dataframe containing 100000 rows as excel. Moments in the execution I received a fatal error : "Python kernel is unresponsive."

However, I am constrained from increasing the number of clusters or other relevant infrastructure change. So my only option is to fix the code. 

I am looking to implement more than one option I would get from this forum. pls suggest.

Thanks

[Ayushi_Suthar]

Hi @chari ,Thanks for bringing up your concerns, always happy to help 😁

We understand that you are facing the following error while you are writing a pandas dataframe containing 100000rows in excel.

As per the Error >>> Fatal error: The Python kernel is unresponsive. The Python process exited with exit code 137 (SIGKILL: Killed). This may have been caused by an OOM error. Check your command's memory usage.

The driver node is OOM leading to this error. This can be fixed by:

• Choosing a higher driver instance size depending on the workload.
• Splitting workloads into multiple clusters
• Moving to a jobs cluster

Please let me know if this helps and leave a like if this helps, followups are appreciated.
Kudos
Ayushi

[chari]

Hello,

Unfortunately, I cant update my cluster until six months. 

But I want to use a spark dataframe to write as a CSV. Does it help?

[Kaniz]

Hey there! Thanks a bunch for being part of our awesome community! 🎉 

We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution for you. And remember, if you ever need more help , we're here for you! 

Keep being awesome! 😊🚀