Unity Catalog Volume mounting broken by cluster environment variables (http proxy)

[Seb_G]

Hello all,
I have a slightly niche issue here, albeit one that others are likely to run into.

Using databricks on Azure, my organisation has included extended our WAN into the cloud, so that all compute clusters are granted a private IP address that can access on-prem servers (using vnet injection). One of those servers is a http/https proxy, through which all our data to non-azure systems should be trafficked. This is achieved through SCC and private vnets.

Recently, to permit the installation of libraries from pypi, I added the following to clusters' environmental variables: http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY. That allowed for installation from pypi on boot when adding the library to the cluster libraries (I am aware that pip --proxy can work in a notebook).

However, I've since discovered that is breaking python's access to volumes mounted in unity catalog. My volume is an azure storage account container. The proxy server is a whitelisted IP in the azure storage account (it's a public ip4). I am account admin, workplace admin, and have manually granted myself all privileges on the catalog level. Compute is a single user compute running DRV 15.4 LTS with config

spark.databricks.cluster.profile singleNode
spark.master local[*]

The below code works as expected regardless of the environmental variables:

path = "/Volumes/<catalog>/<schema>/<volume>/<folder1>/<folder2>/<folder3>/<file>.json"
dbutils.fs.head(path)

The following code functions as expected (reading the file) without the variables set,

with open(path, "r") as fp:
    json.load(fp)

but throws the following error when the proxy variables are set

PermissionError: [Errno 13] Permission denied: '/Volumes/<catalog>/<schema>/<volume>/<folder1>/<folder2>/<folder3>/<file>.json'

I would really like to keep the proxy environment variables to catch all the traffic to the public internet seamlessly (e.g. library installs). What I'm hoping can be answered:

• Why does volume access via dbutils/browsing unity catalog GUI work on the compute when the environmental variables are set, but vanilla python does not?
• Are there any suggested workarounds to allow python to interact with the filesystem when the environment variables are set?
• If I cannot set the http/s proxy environment variables, are there any other variables or spark config that could get the cluster to access pypi/maven etc via a proxy by default?

[amartt]

Bumping this as I am having the same issue.

Is the solution to just not define the proxy vars globally?

Is there something to add in the NO_PROXY or the spark_conf to allow for communication within databricks between storage accounts to not go through the proxy?

Have already tried adding in the storage accounts of use and the databricks workspace to the NO_PROXY, as well as adding the java options for the driver and executor.

[Seb_G]

Unfortunately, the only solution I found was to not use the proxy globally. Good luck!

[amartt · Accepted solution]

A solution that worked, in addition to having the HTTP_PROXY and HTTPS_PROXY variables set globally, was to add the following definition to the compute policy:

"spark_env_vars.NO_PROXY": {

    "type": "fixed",

    "value": "localhost,127.0.0.1,169.254.169.254,*.databricks.azure.com,*.azuredatabricks.net,*.databricks.azure.us,*.databricks.azure.cn,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"

  }

or could just put it straight into the env vars on a cluster itself.