topic How to load xml files with spark-xml ? in Data Engineering

How to load xml files with spark-xml ?

leaw — Fri, 12 Jan 2024 16:34:29 GMT

Hello,

I cannot load xml files.

First, I tried to install Maven library com.databricks:spark-xml_2.12:0.14.0 as it is told in documentation, but I could not find it. I only have HyukjinKwon:spark-xml:0.1.1-s_2.10, with this one I have this error: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: Library resolution failed because unresolved dependency: com.databricks:spark-xml_2.12:0.17.0: not found

Then I tried to install library via dbfs using a JAR file. I tried spark_xml_2_12_0_15_0.jar and spark_xml_2_12_0_17_0.jar, doing this I progressed a little but I have now this error: java.lang.NoClassDefFoundError: scala/$less$colon$less

My cluster Runtime Version is: 13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12)

I have to read my xml files via notebook, thank you in advance for your help.

Re: How to load xml files with spark-xml ?

Lakshay — Fri, 12 Jan 2024 19:47:50 GMT

Hi @leaw , you can install the maven library in your cluster as below:

After that you just need to follow the document: https://docs.databricks.com/en/query/formats/xml.html

Re: How to load xml files with spark-xml ?

leaw — Mon, 15 Jan 2024 12:15:48 GMT

Thanks for your answer. I had already tried this but I have an error as I don't have this library on my databricks.

Re: How to load xml files with spark-xml ?

leaw — Mon, 15 Jan 2024 12:23:17 GMT

I think I have resolved my issue by dowloading and adding last version jar file for scala 2.12, but I don't know if it is a long term solution.

(yesterday it was working then it was not, then yes, it is not very steady.)

If anybody faces this problem, I'll be grateful for sharing experience about reading xml files in databricks.

Re: How to load xml files with spark-xml ?

Lakshay — Mon, 15 Jan 2024 13:49:56 GMT

Hi @leaw , The option I suggested should have downloaded the jar directly from maven but it seems like due to some issue it is unable to download.

Re: How to load xml files with spark-xml ?

Lakshay — Mon, 15 Jan 2024 13:50:29 GMT

Anyway, glad to know that you were able to find an alternate solution.

Re: How to load xml files with spark-xml ?

Frustrated_DE — Wed, 31 Jan 2024 12:03:11 GMT

Hi All,

Installed spark-xml_2.13-0.17.0.jar on runtime 14.2 scala 2.12 and also receiving the error when attempting to read XML. Any advice would be appreciated around how to resolve.

"java.lang.NoClassDefFoundError: scala/$less$colon$less"

Re: How to load xml files with spark-xml ?

Frustrated_DE — Thu, 01 Feb 2024 09:29:39 GMT

Mismatch on Scala version, my bad! Sorted