I am reaching out to bring attention to a performance issue we are encountering while processing XML files using Spark-XML, particularly with the configuration spark.read().format("com.databricks.spark.xml").
Currently, we are experiencing significant processing times, with it taking approximately 35 minutes to process 12,000 files in our AWS EMR environment. Our EMR cluster is configured with 200 vCPUs, 500 GB memory, and 900 GB disk space. We are utilizing the Hudi upsert mode with a total of 30 executors, each with 16.6 GB of memory. However, despite these resources, we are observing suboptimal utilization, as indicated by the attached screenshot, where each executor is only utilizing 4-5 MB.
Upon further investigation, we noticed that the reading phase, during which Spark reads the files or plans to read them, takes approximately 8-10 minutes for the 12,000 files. However, the writing phase, following the processing, takes significantly longer, ranging from 25-30 minutes. We identified that the XmlRecordReader initialize method (referenced here: https://github.com/databricks/spark-xml/blob/v0.14.0/src/main/scala/com/databricks/spark/xml/XmlInpu...) is repeatedly called during the reading phase, which appears to introduce delays in processing paths from S3.
Please let me know if any futher information required
.png){kind=link}