topic Re: Notebook is stuck and cluster goes into waiting state while using spark libraries in Administration & Architecture

Notebook is stuck and cluster goes into waiting state while using spark libraries

BhawaniD — Thu, 12 Sep 2024 10:21:28 GMT

Hey,

We have installed the com.databricks:spark-xml_2.12:0.18.0 library in our VNET-injected Databricks workspace to read XML files from a storage account. The notebook runs successfully for text files when the cluster is started without the library installed. However, when running the notebook with XML files, the cluster enters a waiting state.

Our Databricks subnet has a route table attached, and all traffic is routed through our firewall. When we disassociate the route table from the Databricks public subnet, the notebook runs without any issues, indicating that the firewall is blocking the required connectivity. However, I am unable to determine which ports or FQDNs need to be opened to resolve this issue.

I would greatly appreciate any thoughts on this!

Re: Notebook is stuck and cluster goes into waiting state while using spark libraries

Witold — Thu, 12 Sep 2024 11:13:31 GMT

Since it's a maven dependency it should be simply HTTP and port 80/443.

Besides, are you aware that native XML support is included since runtime 14.3? This replaces the library spark-xml.

Re: Notebook is stuck and cluster goes into waiting state while using spark libraries

BhawaniD — Thu, 12 Sep 2024 11:36:11 GMT

I don't think it is included in the runtime 14.3. I tried running my notebook without installing the library and it fails straightaway because the library has not been installed.

I have opened the firewall rules using service tags on port 443 but still it doesn't help.

Re: Notebook is stuck and cluster goes into waiting state while using spark libraries

Witold — Thu, 12 Sep 2024 12:18:53 GMT

I don't think it is included in the runtime 14.3.

You are wrong here, native XML support is indeed included. Please check the documentation how to use it properly, as there might be slight differences to spark-xml. The reason why it's included is that spark-xml becomes obsolete as it will be part of Spark 4. In Databricks we can use it today starting from version 14.3

I have opened the firewall rules using service tags on port 443 but still it doesn't help.

You might want to consult your network colleagues to configure it properly. Usually maven downloads the libraries e.g. from here (or one of the mirrors)