topic can we read XML files into Dataframes in Spark? in Data Engineering

can we read XML files into Dataframes in Spark?

Srikanth_Gupta_ — Fri, 18 Jun 2021 21:33:41 GMT

Re: can we read XML files into Dataframes in Spark?

Ryan_Chynoweth — Fri, 18 Jun 2021 21:38:55 GMT

Yes, you can use the Spark XML library. https://github.com/databricks/spark-xml

Re: can we read XML files into Dataframes in Spark?

Srikanth_Gupta_ — Fri, 18 Jun 2021 21:39:11 GMT

Yes we can read using below code snippet

val df = spark.read .format("com.databricks.spark.xml").option("rowTag", "<message>") .load("sample.xml")

display(df)

rowTag is important to specify to read the actual content in XML

please read for more details

Re: can we read XML files into Dataframes in Spark?

Mooune_DBU — Fri, 18 Jun 2021 21:39:20 GMT

Yes of course, you can use the

OR use

val df = spark.read
      .format("xml")
      .load("my_file.xml")

More info on the spark-xml api here

Re: can we read XML files into Dataframes in Spark?

sean_owen — Tue, 22 Jun 2021 10:08:46 GMT

Note that you will need to install the spark-xml library to make this work: https://github.com/databricks/spark-xml For example you can create a Library in the workspace that references com.databricks:spark-xml_2.12:0.12.0 and then attach it to a cluster