10-13-2021 07:22 AM
I've created an external table that points to an XML file on the ADLS. I've installed spark-xml library on my cluster. I can query the table on the cluster.
However, I cannot query the table on Databricks SQL. I get the following error. I suppose the spark-xml library should also be installed on the SQL endpoint, however there is no option to do so. Is there any way to install the library on the end point ?
java.util.concurrent.ExecutionException: java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.xml. Please find packages at http://spark.apache.org/third-party'. (35) (SQLExecDirectW)
10-13-2021 08:48 AM
10-14-2021 12:07 AM
Yes. on the Cluster, however the SQL endpoint (cluster for Databricks SQL) does not offer this option.
10-13-2021 09:23 AM
Hello @Bilal Haniff - My name is Piper and I'm a moderator for the community. It's nice to meet you!
Let's see how your fellow community members answer first and then the team will follow up if they need to.
10-15-2021 04:23 AM
@Piper Wilson I believe this is a technical limitation of the platform (Databricks SQL). Can you please check with the team if this is the case?
10-15-2021 09:26 AM
@Bilal Haniff - Yes. It may be next week before I can get back to you. Thank you for your patience.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.