Driver is up but is not responsive, likely due to GC.

[sh_abrishami_ie]

Hi,

I have a problem with writing an excel file into the mounted file.

after 10 mins I see the Driver is up but is not responsive, likely due to GC on the log events.

I'm using the following script:

df.repartition(1).write

 .format("com.crealytics.spark.excel")

 .mode("overwrite") // Optional, default: overwrite.

 .option("header", "true")

 .option("maxRowsInMemory", 1000)

 .save("/mnt/IngestExelFiles/output_fulldf.xlsx")

the feature of the cluster is:

8.4 (includes Apache Spark 3.1.2, Scala 2.12)

Driver type:

85 Gb memory/ 8 cores

the approximate size of the excel file is 200 MB.

would you help to solve this problem?

Thanks

[Hubert-Dudek · Accepted solution]

It is not solution to that problem but I recommend to handle excel reads and writes with Spark Koalas https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.to_excel.html just give it a try maybe it will solve your issue