Databricks Optimization

[Akshay9]

I am trying to read 30 xml files and create a dataframe of the data of each node but i takes alot of time approximately 8 mins to run those files what i can i do to optimize the databricks notebook and i append the data in a databricks delta table 

[Kaniz]

Hi @Akshay9 , Use spark.xml() or com.databricks.spark.xml package: This package provides an efficient way to read and write XML datasets in Spark. It can automatically infer the XML schema and provide additional options for fine-tuning the schema. You can use this package to read the XML data and convert it into a Spark DataFrame.