cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Fatal error when writing a big pandas dF

chari
Contributor

Hello DB community,

I was trying to write a pandas dataframe containing 100000 rows as excel. Moments in the execution I received a fatal error : "Python kernel is unresponsive."

However, I am constrained from increasing the number of clusters or other relevant infrastructure change. So my only option is to fix the code. 

I am looking to implement more than one option I would get from this forum. pls suggest.

Thanks

3 REPLIES 3

Ayushi_Suthar
Honored Contributor
Honored Contributor

Hi @chari ,Thanks for bringing up your concerns, always happy to help 😁

We understand that you are facing the following error while you are writing a pandas dataframe containing 100000rows in excel.

As per the Error >>> Fatal error: The Python kernel is unresponsive. The Python process exited with exit code 137 (SIGKILL: Killed). This may have been caused by an OOM error. Check your command's memory usage.

The driver node is OOM leading to this error. This can be fixed by:

  • Choosing a higher driver instance size depending on the workload.
  • Splitting workloads into multiple clusters
  • Moving to a jobs cluster

Please let me know if this helps and leave a like if this helps, followups are appreciated.
Kudos
Ayushi

Hello,

Unfortunately, I cant update my cluster until six months. 

But I want to use a spark dataframe to write as a CSV. Does it help?

Kaniz_Fatma
Community Manager
Community Manager

Hey there! Thanks a bunch for being part of our awesome community! 🎉 

We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution for you. And remember, if you ever need more help , we're here for you! 

Keep being awesome! 😊🚀

 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!