Bumping this as I am having the same issue.

Is the solution to just not define the proxy vars globally?

Is there something to add in the NO_PROXY or the spark_conf to allow for communication within databricks between storage accounts to not go through the proxy?

Have already tried adding in the storage accounts of use and the databricks workspace to the NO_PROXY, as well as adding the java options for the driver and executor.