can we read XML files into Dataframes in Spark?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-18-2021 02:33 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-18-2021 02:38 PM
Yes, you can use the Spark XML library. https://github.com/databricks/spark-xml
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-18-2021 02:39 PM
Yes we can read using below code snippet
val df = spark.read .format("com.databricks.spark.xml").option("rowTag", "<message>") .load("sample.xml")
display(df)
rowTag is important to specify to read the actual content in XML
please read for more details
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-18-2021 02:39 PM
Yes of course, you can use the
OR use
val df = spark.read
.format("xml")
.load("my_file.xml")More info on the spark-xml api here
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-22-2021 03:08 AM
Note that you will need to install the spark-xml library to make this work: https://github.com/databricks/spark-xml For example you can create a Library in the workspace that references com.databricks:spark-xml_2.12:0.12.0 and then attach it to a cluster