Hi @reachrishav ,

Since 14.3 there is a native support for read and write XML files. Maybe check if it works faster than the library that you've used:

Read and write XML files | Databricks on AWS

And you've mentioned that you write python function to flatten complex types. Do you use it as UDF? Because that could be performance bottelneck also:

What are user-defined functions (UDFs)? | Databricks on AWS