https://community.databricks.com/t5/data-engineering/unittest-in-pyspark-how-to-read-xml-with-maven-com-databricks/m-p/12536#M7336 <P>Please install spark-xml from Maven. As it is from Maven you need to install it for cluster which you are using in cluster settings (alternatively using API or CLI)</P><P></P><P><A href="https://mvnrepository.com/artifact/com.databricks/spark-xml" target="test_blank">https://mvnrepository.com/artifact/com.databricks/spark-xml</A></P> Tue, 26 Jul 2022 12:51:39 GMT Hubert-Dudek 2022-07-26T12:51:39Z https://community.databricks.com/t5/data-engineering/unittest-in-pyspark-how-to-read-xml-with-maven-com-databricks/m-p/12533#M7333 <P>When writing unit tests with unittest / pytest in PySpark, reading mockup datasources with built-in datatypes like csv, json (spark.read.format("json")) works just fine.</P><P></P><P>But when reading XML´s with spark.read.format("com.databricks.spark.xml") in the unit test, this does not work out of the box:</P><P>java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.xml.</P><P></P><P>NOTE: the unit test do NOT run on a databricks cluster, but on a local hadoop winutils directory.</P><P></P><P>Is there any way to implement this, or should I use some python build-in xml libraries?</P> Tue, 26 Jul 2022 09:30:25 GMT https://community.databricks.com/t5/data-engineering/unittest-in-pyspark-how-to-read-xml-with-maven-com-databricks/m-p/12533#M7333 Michael_Galli 2022-07-26T09:30:25Z https://community.databricks.com/t5/data-engineering/unittest-in-pyspark-how-to-read-xml-with-maven-com-databricks/m-p/12534#M7334 <P>I suppose you run spark locally? Because com.databricks.spark.xml is a library for spark.</P><P>It is not installed by default so you have to add it to your local spark install.</P> Tue, 26 Jul 2022 12:10:29 GMT https://community.databricks.com/t5/data-engineering/unittest-in-pyspark-how-to-read-xml-with-maven-com-databricks/m-p/12534#M7334 -werners- 2022-07-26T12:10:29Z https://community.databricks.com/t5/data-engineering/unittest-in-pyspark-how-to-read-xml-with-maven-com-databricks/m-p/12535#M7335 <P>This is correct.. the following worked for me:</P><P>SparkSession.builder.(..).config("spark.jars.packages", "com.databricks:spark-xml_2.12:0.12.0")</P> Tue, 26 Jul 2022 12:49:58 GMT https://community.databricks.com/t5/data-engineering/unittest-in-pyspark-how-to-read-xml-with-maven-com-databricks/m-p/12535#M7335 Michael_Galli 2022-07-26T12:49:58Z https://community.databricks.com/t5/data-engineering/unittest-in-pyspark-how-to-read-xml-with-maven-com-databricks/m-p/12536#M7336 <P>Please install spark-xml from Maven. As it is from Maven you need to install it for cluster which you are using in cluster settings (alternatively using API or CLI)</P><P></P><P><A href="https://mvnrepository.com/artifact/com.databricks/spark-xml" target="test_blank">https://mvnrepository.com/artifact/com.databricks/spark-xml</A></P> Tue, 26 Jul 2022 12:51:39 GMT https://community.databricks.com/t5/data-engineering/unittest-in-pyspark-how-to-read-xml-with-maven-com-databricks/m-p/12536#M7336 Hubert-Dudek 2022-07-26T12:51:39Z https://community.databricks.com/t5/data-engineering/unittest-in-pyspark-how-to-read-xml-with-maven-com-databricks/m-p/12537#M7337 <P>See above, I already found the solution. There is no cluster, but only a local spark session.</P> Tue, 26 Jul 2022 13:19:05 GMT https://community.databricks.com/t5/data-engineering/unittest-in-pyspark-how-to-read-xml-with-maven-com-databricks/m-p/12537#M7337 Michael_Galli 2022-07-26T13:19:05Z