Driver is up but is not responsive, likely due to GC.

[sh_abrishami_ie]

Hi,

I have a problem with writing an excel file into the mounted file.

after 10 mins I see the Driver is up but is not responsive, likely due to GC on the log events.

I'm using the following script:

df.repartition(1).write

 .format("com.crealytics.spark.excel")

 .mode("overwrite") // Optional, default: overwrite.

 .option("header", "true")

 .option("maxRowsInMemory", 1000)

 .save("/mnt/IngestExelFiles/output_fulldf.xlsx")

the feature of the cluster is:

8.4 (includes Apache Spark 3.1.2, Scala 2.12)

Driver type:

85 Gb memory/ 8 cores

the approximate size of the excel file is 200 MB.

would you help to solve this problem?

Thanks

[Kaniz]

Hi @sh.abrishami.ie! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[Hubert-Dudek · Accepted solution]

It is not solution to that problem but I recommend to handle excel reads and writes with Spark Koalas https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.to_excel.html just give it a try maybe it will solve your issue

[Kaniz]

Hi @Shokoufeh Abrishami​ , Can you show the error stack or the logs?