Ayushi_Suthar
Databricks Employee
Databricks Employee

Hi @chari ,Thanks for bringing up your concerns, always happy to help 😁

We understand that you are facing the following error while you are writing a pandas dataframe containing 100000rows in excel.

As per the Error >>> Fatal error: The Python kernel is unresponsive. The Python process exited with exit code 137 (SIGKILL: Killed). This may have been caused by an OOM error. Check your command's memory usage.

The driver node is OOM leading to this error. This can be fixed by:

  • Choosing a higher driver instance size depending on the workload.
  • Splitting workloads into multiple clusters
  • Moving to a jobs cluster

Please let me know if this helps and leave a like if this helps, followups are appreciated.
Kudos
Ayushi