Driver is up but is not responsive, likely due to GC.

sh_abrishami_ie — Wed, 12 Jan 2022 09:37:42 GMT

Hi,

I have a problem with writing an excel file into the mounted file.

after 10 mins I see the Driver is up but is not responsive, likely due to GC on the log events.

I'm using the following script:

df.repartition(1).write

.format("com.crealytics.spark.excel")

.mode("overwrite") // Optional, default: overwrite.

.option("header", "true")

.option("maxRowsInMemory", 1000)

.save("/mnt/IngestExelFiles/output_fulldf.xlsx")

the feature of the cluster is:

8.4 (includes Apache Spark 3.1.2, Scala 2.12)

Driver type:

85 Gb memory/ 8 cores

the approximate size of the excel file is 200 MB.

would you help to solve this problem?

Thanks

Re: Driver is up but is not responsive, likely due to GC.

Hubert-Dudek — Sat, 15 Jan 2022 15:46:21 GMT

It is not solution to that problem but I recommend to handle excel reads and writes with Spark Koalas https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.to_excel.html just give it a try maybe it will solve your issue