Hi All,

Recently we have implemented the change to make databricks workspace accessible only via a private network. After this change, we found lot of errors on connectivity like from Power BI to Databricks, Azure Data factory to Databricks etc.

I was able to resolve above mentioned issues whereas currently I am stuck with some other issue whose root cause looks like the same implementation.

The Error I get is:

run failed with error message Library installation failed for library due to user error for whl: "dbfs:/Volumes/any.whl" Error messages: Library installation attempted on the driver node of cluster XXXX and failed. Please refer to the following error message to fix the library or contact Databricks support. Error code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error message: java.util.concurrent.ExecutionException: org.apache.spark.sql.AnalysisException: 403: Unauthorized access to workspace: 12345543234

I have executed this job for 200 times and out of which it failed with above error on 13 instances. So, it is clearly a temporary issue happenning sometimes.

Does someone have faced similar issue or has any idea how to proceed with this?