Databricks Runtime, Pyspark and Spark Versions

loujiang
New Contributor II

Hello, Dear community,

I was go through the documentation of function from_xml here pyspark.sql.functions.from_xml — PySpark 4.1.2 documentation, it denotes that it is available in pyspark version higher than 4.0.0. 

Meanwhile, we have documentation for from_xml at Azure/ AWS,from_xml function - Azure Databricks - Databricks SQL | Microsoft Learn the support of it is above Databricks Runtime 14.1. 

But the Databricks Runtime 14.1 are using Apache Spark version 3.5.0, which should has no from_xml implementation. How should we understand this difference?

Thanks

best wishes

loujiang