How to install JAR libraries from ADLS? I'm having an error

[oussamak]

I mounted the ADLS to my Azure Databricks resource and I keep on getting this error when I try to install a JAR from a container:

Library installation attempted on the driver node of cluster 0331-121709-buk0nvsq and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: java.util.concurrent.ExecutionException: Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key

This is the path I provide to Databricks: abfss://libraries@nespdatalakeadls2.dfs.core.windows.net/libraries/tensorflow-1.15.0.jar

[Kaniz_Fatma]

Hi @Oussama KIASSI​ ,

The error message says :-

Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key

You can't use the storage account access key to access data using the abfss protocol. You need to provide more configuration options if you want to use abfss- it's all described in the documentation.

Storage access key could be used only when you're using wasbs, but it's not recommended to do with ADLSGen2.

P.S. You can also use the passthrough cluster if you have permission to access that storage account.

[oussamak]

Hi @Kaniz Fatma​,

Thank your reply. I followed the steps in the link to mount the ADLS from the notebook but I'm getting an error :

Operation failed: "This request is not authorized to perform this operation.", 403, HEAD, https://myadls2.dfs.core.windows.net/libraries/?upn=false&action=getAccessControl&timeout=90
ExecutionError                            Traceback (most recent call last)
<command-274923613237744> in <module>
      5           "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/12a3af23-a769-4654-847f-958f3d479f4a/oauth2/token"}
      6 
----> 7 dbutils.fs.mount(
      8   source = "abfss://libraries@myadls2.dfs.core.windows.net/",
      9   mount_point = "/mnt/libraries",
 
/databricks/python_shell/dbruntime/dbutils.py in f_with_exception_handling(*args, **kwargs)
    387                     exc.__context__ = None
    388                     exc.__cause__ = None
--> 389                     raise exc
    390 
    391             return f_with_exception_handling
 
ExecutionError: An error occurred while calling o360.mount.
: Operation failed: "This request is not authorized to perform this operation.", 403, HEAD, https://myadls2.dfs.core.windows.net/libraries/?upn=false&action=getAccessControl&timeout=90
	at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:246)
	at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:955)
	at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:937)
	at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:318)
	at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:883)
	at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:682)
	at com.databricks.backend.daemon.dbutils.DBUtilsCore.verifyAzureFileSystem(DBUtilsCore.scala:824)
	at com.databricks.backend.daemon.dbutils.DBUtilsCore.createOrUpdateMount(DBUtilsCore.scala:734)
	at com.databricks.backend.daemon.dbutils.DBUtilsCore.mount(DBUtilsCore.scala:776)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
	at py4j.Gateway.invoke(Gateway.java:295)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:251)
	at java.lang.Thread.run(Thread.java:748)

FYI: The service principal has the following roles on the adls : storage data reader, storage blob data contributor, storage blob data owner, owner.